New AI Model Fills in Blank Spots in Photos (nikkei.com)

A new technology uses artificial intelligence to generate synthetic images that can pass as real. From a report, shared by a reader (the link may be paywalled): The technology was developed by a team led by Hiroshi Ishikawa, a professor at Japan's Waseda University. It uses convolutional neural networks, a type of deep learning, to predict missing parts of images. The technology could be used in photo-editing apps. It can also be used to generate 3-D images from real 2-D images. The team at first prepared some 8 million images of real landscapes, human faces and other subjects. Using special software, the team generated numerous versions for each image, randomly adding artificial blanks of various shapes, sizes and positions. With all the data, the model took three months to learn how to predict the blanks so that it could fill them in and make the resultant images look identical to the originals. The model's learning algorithm first predicts and fills in blanks. It then evaluates how consistent the added part is with its surroundings.

Comments

It might be ok and awful at the same time.

[Mal-2]

I bet it will be pretty good in some contexts, and most likely an improvement overall compared to content-aware fills. However, when it completely falls on its face I bet it will be even funnier than the way content-aware fill blows up. Lower rate of occurrence, but much more hilarity when it happens.

Re:It might be ok and awful at the same time.

[PolygamousRanchKid]

However, when it completely falls on its face I bet it will be even funnier than the way content-aware fill blows up.

I tried the prototype, and it filled all the holes with pictures and outrageously fake quoted statements from Donald and Hillary. The system must have been hacked by Russians already.

The pictures will probably be updated really soon with pictures of the new presidential candidates, Oprah Winfrey and Zuckerberg.

Re:It might be ok and awful at the same time.

[ShanghaiBill]

when it completely falls on its face I bet it will be even funnier than the way content-aware fill blows up.

Just put the failures into the training set, re-run backprop, and deploy the upgraded NN.

Alternate Headline

[OzPeter]

The alternate headline is:

Computer program analyzes data and based on that analysis invents new data that seems plausible to most people

Re: Alternate Headline

[javaman235]

I think it's really deeper tho. If you see just the tip of a football an NFL running back is holding, your brain has a plausible idea that he's holding a football, (not a puppy with a butt that looks like a football) and that will be confirmed when he spikes it in the end zone. Building most probable world models from partial information is a key thing human intelligence does.

Re: Alternate Headline

[OzPeter]

I think it's really deeper tho. If you see just the tip of a football an NFL running back is holding, your brain has a plausible idea that he's holding a football, (not a puppy with a butt that looks like a football) and that will be confirmed when he spikes it in the end zone. Building most probable world models from partial information is a key thing human intelligence does.

Nice straw man bro!

There is a difference between inferring that a complete football is attached to the tip and creating (from scratch) an image of the hidden parts of the football. Because until you actually see it, the hidden parts of the football may well be painted to look like a puppy.

As in the case of TFA where they created from scratch a shelf full of books without knowing if that shelf actually contained books or a bust of Plato. And in fact from all we know of the "before" image presented in the TFA there might have been a bust of Plato on the shelf - or even no shelf at all.

Re: Alternate Headline

[javaman235]

The feeling of surprise is when your brain encounters something different from the world model *it already built* from partial information and held subconsciously. It's a part of intelligence.

I mean yeah, if someone peddles this is purely accurate info it's BS, but you can't deny that statements about what's probably in a concealed area, based on lots of experience/training are useful if taken for what they are.

Can it capture the whole boob?

[Anonymous Coward]

Many celebrities like to show off underboob, side boob, cleavage and the occasional nip slip.

Does this software allow us to piece together a whole CELEBRITY BREAST?

Re:Can it capture the whole boob?

[Anonymous Coward]

that's the spirit of old slashdot.

Good to see it back.

Re: Can it capture the whole boob?

[Anonymous Coward]

They are ugly as shit. Better off keep them away from sight.

Re: Can it capture the whole boob?

[Anonymous Coward]

dunno, perhaps you should booble it?

Re:Can it capture the whole boob?

[sysrammer]

that's the spirit of old slashdot.

Good to see it back.

Yeah...

I can't wait 'til the internet teaches the AI about nazis and racism and stuff.

No possible way to be used for fraud. Nope.

[mark-t]

Honestly, that's the first application I thought of if or when this tech becomes commonly available.

Where it gets more scary is when it used to manipulate pictures that may be used in the future to ascertain the veracity of particular circumstances or events, possibly even for legal reasons.

Interlacing? Special effects scaling?

[Anonymous Coward]

I wonder how well these techniques could be applied to interlaced material or whether it could be used to scale up special effects (e.g. Voyager). Since interlacing destroys information in some domain, being able to generate information that fits seems a natural use case. Scaling up special effects could work upon the same principle.

Re:Interlacing? Special effects scaling?

[Hal_Porter]

It's just so dense

Re:Interlacing? Special effects scaling?

[ezdiy]

Yes, simple GPU NN filters are routinely used for upscaling and deinterlacing (fe. nnedi3 in madvr) these are coarse, but work in real time.

Filling missing parts of image is vastly more difficult problem though, and is very unlikely to work in real time - so it is useless for deinterlacing or upscaling.

Useless

[Anonymous Coward]

Photos and videos are becoming so easily manipulated that they will soon be useless as proof of anything.

Re:Useless

[OzPeter]

Photos and videos are becoming so easily manipulated that they will soon be useless as proof of anything.

This is nothing new. Photos have been manipulated from the time of their invention. The very act of pointing a camera in a particular direction manipulates the image and hence proof of something.

Re:Useless

[omnichad]

Security cameras need verifiable cryptographic signatures on both the timestamp and the frame contents right now. Yes, those can still be faked, but not as easily as image manipulation.

Re:Useless

[Maxo-Texas]

But if they can be faked, then you can't trust them.

You'll need multiple independent video sources (like different people with different camera's).

The example photo sucks- it's obvious where it was filled. I bet it would be very obvious for video footage but I'm sure the tech will improve further over time.

Re:Useless

[ShanghaiBill]

Security cameras need verifiable cryptographic signatures on both the timestamp and the frame contents right now.

Says who? My company has security cameras, and I can assure you that they do not encrypt anything. Are we breaking the law?

Amazon sells dozens of security cameras. None of the descriptions mention any encryption other than for the Wifi connection.

Re:Useless

[HiThere]

He didn't say encrypted, but rather cryptographically signed. That's essentially a modified hash of the picture, or well-defined parts of the picture. It generally wouldn't be visible if you were looking at the picture.

OTOH, I also doubt that the general run of cameras used for security do this signing. I suspect that he had a specific variety (not brand) of camera that he was thinking of when he said "security camera" which is a lot different from the normal web-cam that is mounted on a porch.

Re:Useless

[Anonymous Coward]

Security cameras need verifiable cryptographic signatures on both the timestamp and the frame contents right now.

Ha, maybe in sci-fi movies they do.

Re:Useless

[Cederic]

I interpreted him as suggesting that non-repudiation of video footage is now fucking hard to achieve and needs building into the initial capture.

Re:Useless

[omnichad]

This. And there are few products even on the market yet, if any, that are doing this. I don't think anyone realized how soon this would be needed and nobody is willing to pay for it until it is.

Re:Useless

[sysrammer]

In ancient times it used to be hard to do video spoofing, for any lengthy footage. I remember Forrest Gump being the first time that I really got the message that, just like photos, a video of an event will no longer be sure proof.

What's the saying? Don't believe anything you hear, and only half of what you see?

Put the heads back in?

[Anonymous Coward]

One of your psycho ex's or relatives cuts all the heads out of your family photos, this program puts them back in. Or maybe we use deepfake for that!

Not bad, but not great

[lurker412]

The example shown in the linked article doesn't hold up under scrutiny. Look at the blue-green books on the center-right--the convergence of the shelves is wrong and the corner is not rendered correctly. Assuming this was a one-step edit, it's probably better than Photoshop's current content aware fill, but it still requires additional work to escape detection.

Re:Not bad, but not great

[epine]

The example shown in the linked article doesn't hold up under scrutiny.

The story submission did not say the approach was good enough to evade focussed scrutiny.

Confucius Says: What was never held up is not able to not hold up. Then he pats himself contentedly on the belly.

Might look real

[nospam007]

...but not necessarily _be_ real.

Depending on what's missing of a photo of Long John Silver, it might create one with 2 parrots and 2 eye flaps.

Anti-shake

[bill_mcgonigle]

Hopefully this can be applied to anti-shake filters where existing solutions do a really poor job of inventing blurry missing data.

Re:Anti-shake

[Solandri]

Hopefully this can be applied to anti-shake filters where existing solutions do a really poor job of inventing blurry missing data.

The data is actually still there, not missing. It's just smeared across several pixels instead of each point in space corresponding to a single pixel. You can use a deconvolution filter to un-smear it. Same goes for out-of-focus photos.

In the case of camera shake, if the camera would record its movements while the photo was being taken and included that info in the photo, you could apply a perfect deconvolution filter that would almost completely eliminate the blur (it becomes less accurate near the edges because you've permanently lost info when parts of the photo moved out of the frame).

Photoshop CC already includes this filter. I suspect the only reason it isn't yet built into phones to automatically de-blur photos with camera shake is because it's fairly processor intensive, making it easier just to take another photo. (Samsung's strategy is to take multiple photos held in a temporary memory buffer, then the phone determines which is sharpest, and lets you choose if you want to keep it, then deletes the others.)

Re:Anti-shake

[wonkey_monkey]

In the case of camera shake, if the camera would record its movements while the photo was being taken and included that info in the photo, you could apply a perfect deconvolution filter that would almost completely eliminate the blur

It's easier and probably more robust to compensate by moving the sensor.

Re:Anti-shake

[Hal_Porter]

In the case of camera shake, if the camera would record its movements while the photo was being taken and included that info in the photo, you could apply a perfect deconvolution filter that would almost completely eliminate the blur (it becomes less accurate near the edges because you've permanently lost info when parts of the photo moved out of the frame).

That's a really excellent idea. Best thing is that if we started doing that now with an imperfect filter later on as filters improved the images would get sharper so long as you had access to the original image with the 'acceleration as a function of time' data in it.

It's almost as good as Bladerunner's 'Zoom. Enhance' image processing.

Re:Anti-shake

[Hal_Porter]

Why not do both? You could use acceleration data to do OIS but at the same time store the acceleration data in the image so that in the future when image processing improved you could get an even clearer image.

Please, someone make a plug-in...

[Anonymous Coward]

...or add that to VLC to remove the bugs and popup ads that network TV adds to all this content. Ihateit, ihateit, ihateit, ihateit, ihateit, I HATE IT!

Meh

[Anonymous Coward]

Deep Image Prior could do inpainting without a training set, and by training in parameter-space only.

Re:Meh

[wonkey_monkey]

"Meh. Something superficially similar was done before therefore this is not noteworthy."?

Great logic.

Dyslexia in adults...

[Gravis Zero]

I saw these in my RSS feed...

US's Greatest Vulnerability is Ignoring the Cyber Threats From Our Adversaries, Foreign Policy Expert Says
New AI Model Fills in Blank Spots in Photos

and misread the second as...

New Al Gore Fills in Blank Spots in Photos

I was very disappointed.

Comment removed

[account_deleted]

Comment removed based on user account deletion

Now I can unblur...

[CaptainDork]

... the Trump statues.

Re:Now I can unblur...

[CaptainDork]

Too soon?

Show the original

[Anonymous Coward]

It would have been useful if TFA had also shown the original photo without the elisions just to see how close the restoration came.

Re:Show the original

[HiThere]

I think you're missing the point. The point is not to do an accurate reconstruction of missing information, but to do a plausible one. If the data is missing, this approach cannot regenerate it.

Re:Show the original

[crunchygranola]

While it would have been somewhat interesting, the image fill in does not look normal at all. It is blurred, poorly colored, and distorted - like a bad photoshop. If one had been shown a stack of photos that had not been altered, and this one stuck in, and were asked to find the altered photo I don't think you would have had any problem picking this out.

Algorithms and People

[crunchygranola]

Algorithms and people process images differently. This AI/ML technique creates plausible fill that will fool most people, most of time.

Any Slashdot regular has seen the posting about image manipulation techniques that can fool a neural net image recognition system to think an AK-47 is bunny rabbit, or vice versa. But the manipulated images look pretty much the same to people. Why do the altered images get misidentified? We can;t really say, given that we can;t really say why the neural net identifies an unmodified image correctly in the first place, it just learned to. People on the other had can point out the parts of an AK-47 or a bunny rabbit.

In this case I strongly suspect that no matter how much the fake fill in looks like real data, a suitable algorithm can detect the manipulated part of the image from the original image. There are many kinds of statistical tests that could be performed, and while a clever faker might be able to anticipate and "fix" some of them, it would be another matter entirely to be able to fix all possible ones.

This is similar to the effective impossibility of a forger to write something supposedly written by someone else so that the forgery cannot be detected. You can copy style and language up to a point, but you can't mimic every feature that can be analyzed statistically, nor can you conceal all of you own habitual traits.

It will be a good while yet before images can be altered in such a way that the fact of alteration cannot be detected. And as other note here, embedded cryptographic hashes can make images effectively secure even from that.

The story is different of course if what you are dealing with is the lossy compression of an image.

Content Aware

[darkain]

Content Aware Fill has been around for over a decade now. Adobe acquired a small team that worked on it in the early 2000s, and I believe Photoshop CS4 was the first version to use the tech as Content Aware Resize. CS5 included Content Aware Fill (which is what this article is describing it seems). These were released in 2008 and 2010 respectively.

Re:Content Aware

[wonkey_monkey]

Umm, okay? This is just a bit of a step-up from Photoshop's Content-Aware Fill.

I presume, to you, that the fax machine was nothing but a waffle iron with a phone attached.

Re:Content Aware

[Goragoth]

No, this is nothing like the content-aware algorithms used in Photoshop (the function is the same but the underlying technology is radically different). A content-aware algorithm simply analyses what is left of the picture and uses that information to guess what should go into the missing parts. The algorithm described here is a deep learning system trained on thousands of similar images. It can then use this data to help it guess how to fill in the missing pieces. This means it is capable of data synthesis that is not present within the image being processed. That's huge.

All that said the example picture shown in the article had some rather obvious flaws. If that's a best-case outcome then I think they still have some work to do frankly (might just need training on more data).

One image and no further links? Is that it?

[wonkey_monkey]

The extremely shorted linked article has a single, low-res, unzoomable image, and no link to any further information. There must be a better source than this.

Bubble your photos is soooo 2018

[Anonymous Coward]

There must be someone on Reddit who has in interest in this.

We need this and it's relative

[pecosdave]

I've wanted to come up with an AI software for old Academy Ratio movies like the Wizard of Oz.

If a camera pans, use leading and trailing info to create edges.

If it's a still shot use something like this to generate probable edges.

Make Academy ratio movies 16:9 and do the same to some old TV shows. If I could pull this off I could potentially be the most hated man since the onset of colorization of old movies....