Ask Slashdot: Tips On 2D To Stereo 3D Conversion?

An anonymous reader writes "I'm interested in converting 2D video to Stereoscopic 3D video — the Red/Cyan Anaglyph type in particular (to ensure compatibility with cardboard Anaglyph glasses). Here's my questions: Which software(s) or algorithms can currently do this, and do it well? Also, are there any 3D TVs on the market that have a high quality 2D-to-3D realtime conversion function in them? And finally, if I were to try and roll my own 2D-to-3D conversion algorithm, where should I start? Which books, websites, blogs or papers should I look at?" I'd never even thought about this as a possibility; now I see there are some tutorials available; if you've done it, though, what sort of results did you get? And any tips for those using Linux?

Comments

Here's a tip

[Hatta]

Don't do it.

Re:Here's a tip

[spidercoz]

Exactly. People with the proper equipment and money fail at this regularly.

Re:Here's a tip

[ciderbrew]

Unless you only have one eye. I'm sorry to tell you ...

Re:Here's a tip

[vlm]

Exactly. People with the proper equipment and money fail at this regularly.

I think it is important to arrive at the correct mindset. This has never stopped people from snapping pix at weddings and sporting events and tourist traps, even if their pix look like garbage compared to a pro photo on a postcard or whatever.

If you want to do it for fun, heck yes go for it. Go Go Go. You don't need help just try it.

If you think you'll turn out something that means anything to anyone else in the world, you'll probably be disappointed. Insert stereotype of goans when someone wants to show you old fashioned slides of their vacation. Although that old tech is getting kind of retro cool now.

Re:Here's a tip

[gfxguy]

You're in luck... Peter Jackson is pushing 48fps over 24 in the cinemas, stating enough digital projectors are capable. He's shooting the Hobbit at 48fps, and shooting it in 3D from the get-go. I'm more interested in the content of the movie, but I'm expecting it'll be one of the best, if not THE best, attempts at 3D so far (Jackson Explains "Hobbit" 48FPS Shooting).

He's trying to encourage future film productions to step up to 48, too.

Re:Here's a tip

[Jarik C-Bol]

did you know they storyboarded the hobbit in 3D? saw a bit about it, they had two storyboard artists, sitting side by side, one drew in cyan, the other in red, and they both had glasses available, so the resulting storyboards where in 3D. really kind of an amazing co-op effort on the artists part. (saw a thing about it on the production vlogs, had to break out my cyan/red sunglasses for that episode)

Re:Here's a tip

[Jappus]

Don't do it.

Well, there is a way to do it, a very elegant way even. One that can be, for all purposes and intents, as good as you can get with the raw material; even to the point where the average human will not be able to tell the difference.

The thing is: That solution has a big catch. How big? Well, to put it mildly, you will most likely win the Turing Award in the process of doing so and will at some point end up with a Nobel Prize in your hand, too. As you can imagine, the solution is: Artificial Intelligence; and if you want to really do it, only strong artificial intelligence will do.

The fact is, as others have quite succinctly pointed out, that the issue is in determining what is "in front" and what is "in the background" on top of how far away everything is. This is, quite simply, impossible to do right if you approach it as a purely algorithmic picture-to-picture problem. There is just not enough information inside the frames/movie to do it well enough even at the best of times.

So, what do you do? Easy, you import external information. Things like: "This is a tree; That is a human. A tree is bigger than a human. Both take up the same space in the picture. Assumption: The human is closer than the tree. Proof: The tree casts a shadow on the human and the only light source is behind the tree. Angles point to a distance of 20 meters between human and tree. Etc. pp."

This line of reasoning imports lots of information from the outside; essential things like "What does a tree/human look like?", and "What are their relations to each other size-wise?". But if you grant that this information can be derived and used by an AI, the result can be a very precise derivation of the distances between objects.

It is exactly the same line of reasoning the human brain uses for large distances (where the parallax of your eyes is too small, focus is unimportant and difference between eye positions negligible), or when you have lost vision in one eye (or just plainly covered it). Even though your brain suddenly has only half the information, it is capable of giving you a good feeling for distance and depth.

Of course, it doesn't always work, as far too many optical illusions like the Ames-Room show, but it works significantly better than a "pure" picture-to-picture approach and is the sole reason why almost everyone here feels that 2D-3D conversions are so horrible:

Their brain tells them, that what they see just can't be correct, even if their eyes have actually seen it.

But of course, just using 2 cameras is much simpler. So good luck with (strong) AI. I would be surprised if you solved this issue all by yourself. :)

Re:Here's a tip

[diamondmagic]

I may be nitpicking, but ugh it's important... Cameras are only 2D. For good stereoscopic output, you need a stereoscopic camera. The recent "3D movies" are really called stereoscopic in the industry, "3D" is strictly a marketing term. 2D to 3D conversion, which adds a Z-buffer (the third dimension), still outputs a stereoscopic image in the end (which would be 3D stereoscopic, being generated from a 3D model).

Re:Here's a tip

[NorQue]

Now, there's plenty of automatic algorithms that already improve this in popular videoprojectors and TV sets, but I haven't experienced it first hand so I can't vouch for it.

Be glad you haven't seen that yet. Those "upscale" (for the lack of a better word for it) algorithms make the picture look absolutely horrible, much like a cheaply produced VHS Porno. Now, there's nothing wrong with pr0n, but I'm not especially a fan of the cheap 80s porn esthetic. I'd rather scratch my eyes out than watching a complete movie that way.

new generation, new suckers for '3d'

[TheGratefulNet]

we all were suckered. we tried it, hated it and moved on.

each time they try to re-invent this, its still just an effects gimmick.

you'll soon grow bored.

don't invest anything in this. its a reocurring cash grab due to industry boredom.

and as a fulltime glasses wearer, I'd never be caught dead with cardboard glasses over my regular ones. an absurd concept if there ever was one.

Re:new generation, new suckers for '3d'

[spidercoz]

and as a fulltime glasses wearer, I'd never be caught dead with cardboard glasses over my regular ones. an absurd concept if there ever was one.

Word. Pay twice as much to wear ANOTHER pair of glasses and watch something that will more than likely give me a headache? Where do I sign up!

Until they have fully immersive holography, count me out.

Re:new generation, new suckers for '3d'

[LiquidLink57]

It seems its decision is based upon movement. The more an object moves, the more it jumps out. It is logical, but you can imagine where that doesn't work.

That's what comes from basing their 3D algorithm on T-Rex vision.

This sounds familliar

[localman57]

I'm interested in converting 2D video to Stereoscopic 3D video

George Lucas, is that you?

Special FX

[damaki]

A friend of mine used to work for a French special effects company and he had to work on this. He told me that this is basically a world of pain and it produces great piles of smocking shit. It just sucks, even when done properly by highly trained people. Can you imagine making 3D out of a 2D tree? Make every background 3D or properly cut out the character to get the desired effect?
It sucks, it's mostly manual, get over it.

Waitaminute...

[CanHasDIY]

That's you, isn't it George Lucas?

Dammit, leave the original trilogy alone! The digital "remaster" was insulting enough!

"Arduino sucks"

[sideslash]

Clarification -- Arduino doesn't suck, just paraphrasing the unfortunate mentality of a bunch of posters on this article. It is bewildering to me that on a "news for nerds" site, people are disparaging somebody from undertaking what could turn out to be a cool tech project, even if it is known in advance that the end result isn't going to be "Avatar". And even if the best of 3D is a bomb in the theater, that doesn't mean it isn't a lot of fun to play with, as a school project, etc. I enjoyed messing with this stuff in physics lab in college.

Contra my provocative subject, Arduino is an excellent choice for serious hobbyists. And similarly, there is nothing wrong with playing around with 3D video techniques and even being willing to try rolling one's own algorithm.

Get a (homebrew friendly) life, slashdotters!

(If the OP clarifies that he's working on a big Hollywood title, I'll take this back. Until then...)

this can be done easily with ffmpeg and imagemagic

[nitrofurano]

this can be done easily with ffmpeg and imagemagick - you need two video sources, and from a ffmpeg script, extracting a picture sequence from both videos, one sequence from the left camera, and another from the right - with a bash script using imagemagick you will separate the colour channels from each frame: red from one camera, and green/blue from another - and having the separation done, you will join with imagemagick again the red channel picture frame from one and green/blue from another, into a new picture sequence, and when you have this sequence ready, you convert it into video again with ffmpeg - try googling for ffmpeg and imagemagick instruction arguments when coding this bash script

Re:You're missing critical information

[sideslash]

Let's say you have a video camera poked out of the side window of your car, and you're driving down a road alongside a wide field. The field is sparsely populated with trees, and there are mountains far off in the background.

With the use of video in such a case, the depth information can be pretty accurately inferred from the parallax effect, due to the fact that your car (and camera) are moving along the road. It's a difficult problem, but by comparing frame with frame, an algorithm might piece a somewhat reasonable stereoscopic render of such a scene. There are many other scenes where that approach is futile, but your assertion that all depth is lost is not accurate for the case of (some) video (under ideal conditions). Let's not oversimplify the issue.

Creating "3d"

[Maximum Prophet]

There was a recent NOVA episode about aerial photo reconnaissance during WWII. To make stereoscopic images, they'd fly the plane straight and level over the target. If they could take multiple pictures with 60% overlap, they could use two adjacent images to make one stereoscopic image that was good enough to tell a ship from a decoy.

Any motion picture where the camera pans side to side gives an opportunity to create a "3d" image. If an object moves across a still camera, you can also derive 3d information. (Also if it spins)

An interesting exercise would be to process a film, and make stereoscopic only what what can be done properly, and leave the rest flat. A scene would start out flat, then people and things would begin to jump out at you.

Re:Creating "3d"

[camperdave]

It depends on how they accomplished the pan. If it was by pivoting the camera on the stationary tripod, then it won't work. If it was by laterally moving the camera on a dolly or crane, then you've got something.

How to convert your 2D display into 3D

[backslashdot]

1. Display 2d images on a flat panel tv facing you
2. spin the display 45 degrees so that one edge is nearer to you the other edge
3. That's it --notice how pixels on one side are closer to you when the ones on the opposite edge are futher away from u spetially)you display is in 3D now.

2D to 3D Algorithms

[r2r2]

I have not seen many replies on algorithms, so here is what I know from a researcher point of view.

In a few words: if you only have a 2D video, then it is a very hard computer vision problem, that has not been solved on the research side.

There is an active benchmark of disparity estimation algorithms (full bibliography at the end of the page). Those algorithms take two pictures and estimate a depth image. From this depth image, it is possible to reconstruct the scene in 3D (but you cannot see what's behind objects). From my experience, this class of algorithms do quite a bad job with real-life images, and have not been applied to video at all.

I've been using optical flows (see a related benchmark) for the development of an Android app (3D Camera) that converts pictures from 2D to 3D, without glasses (check it out!). The optical flow is a more general version of depth estimation (i.e. in any direction, not just left to right motion motion). It has been applied 3D conversion of videos with relative success, I can search for references if you are interested.

From my knowledge & experience, optical flows are the state of the art algorithms to convert 2D pictures/videos to 3D, but they are quite computationally intensive.

Monoprice 2D to 3D HDTV/DLP Converter

[JasonRabbit75]

Monoprice sells a 2D to 3D HDTV/DLP Converter (Frame Sequential, Side by Side, and Red/Cyan) w/ Remote for $95

Re:You're missing critical information

[camperdave]

There are a host of techniques to apply depth to a scene. Parallax from multiple camera angles is one. Vanishing point analysis is another. Prototype mapping (a human is going to be *this* shape, with *these* depths) and size of motion analysis (big motions are likely to be caused by objects closer to the camera) may also help.

However, the easiest way is to just shoot the thing in 3D in the first place.