Open Source Camera For Computational Photography

David Orenstein writes "Stanford Computer Science researchers are developing Frankencamera, an open source, fully programmable and finely tunable camera that will allow computational photography researchers and enthusiasts to develop and test new ideas and applications — no longer limited by the features a camera manufacturer sees fit to supply. Disclosure: The submitter is a science writer for Stanford and wrote the linked article."

Comments

Listen up camera manufacturers

[QuantumG]

Please make a camera with:

1. A built in clock that actually keeps time.
2. Built in GPS.
3. Some sensible connectors to upload videos in real time using appropriate external devices, or,
4. Built in Wifi/3G.
5. And all the good camera stuff.

In one device. Oh, and if you can actually make a scanning range finder at a sensible price and embed that too, that'd be great.

Re:Listen up camera manufacturers

[quanticle]

And what exactly do you mean by *good* camera stuff? I mean, not everyone needs (or wants) 12 megapixel full-frame sensors. For the vast majority of (not-professional) shooting, the sensors in DSLRs these days are overkill.

Personally, I'd rather have a point and shoot in my pocket (meaning I can actually use it) versus a super-expensive DSLR that always gets left at home due to bulk or concerns about damage.

Re:Listen up camera manufacturers

[BadAnalogyGuy]

Most of those things aren't software-related at all.

How would Open Source help?

Re:Listen up camera manufacturers

[supernova_hq]

Because an open source phone would most likely have open source, easily upgradable circuits so you could add the devices yourself and KNOW they will work. You could even make your OWN devices and hook them up to it, then write your own driver and have them configurable just like everything else built into the camera.

Re:Listen up camera manufacturers

[Entropius]

Then you want something like the Panasonic LX3.

There are also small-ish DSLR's and DSLR-likes that are a far cry from the full-frame beasties. See Olympus E-620 (a small DSLR), or any of the Micro Four-Thirds cameras.

You can have good image quality and optics along with small these days.

Re:Listen up camera manufacturers

[mrmojo]

I hear you - we want the same thing. Our target is basically a Nokia N900 (which covers 1, 2, 3, 4 and runs linux to boot), plus a much higher quality sensor and lens.

- Andrew (one of the grad students working on the project).

Re:Listen up camera manufacturers

[PaintyThePirate]

None of the cameras you suggested would fit in a sane person's pocket. Pocket sized implies something more along the lines of a Canon Powershot, complete with a tiny sensor and mediocre optics.

Though, with CHDK, you can do some nifty things with them.

Re:Listen up camera manufacturers

[Miseph]

People who have reason to believe that they will want to both take pictures AND be somewhere that cannot obtain GPS data at the same time. Like inside of a cave, or a large reinforced concrete building (such as those in which many photo studios are located).

I'm pretty sure that covers virtually everyone, and I'm also pretty sure that the timekeeping chip from a $10 digital wristwatch would pretty well do the trick.

Re:Listen up camera manufacturers

[Anonymous Coward]

And what exactly do you mean by *good* camera stuff? I mean, not everyone needs (or wants) 12 megapixel full-frame sensors. For the vast majority of (not-professional) shooting, the sensors in DSLRs these days are overkill.

Personally, I'd rather have a point and shoot in my pocket (meaning I can actually use it) versus a super-expensive DSLR that always gets left at home due to bulk or concerns about damage.

Its overkill until you want to take a decent picture in really bad lighting without a flash.

Re:Listen up camera manufacturers

[callinyouin]

Um, we also use them to snap shots of boobs at parties.

Re:Listen up camera manufacturers

[Cheesetrap]

Its overkill until you want to take a decent picture in really bad lighting without a flash.

Stop peeping in her window bro, that's not cool.

Re:Listen up camera manufacturers

[RuBLed]

The size and weight difference between those mediocre Powershots and LX3 is almost negligible. LX3 is still pocketable. I could say that LX3 is a superb camera vs those mediocre Powershots. Powershot G10 would be the closest competitor during their launch last year but most agree that the G10 would be harder to pocket than an LX3.

I own an LX3 and had seen/felt a G10. I bring my LX3 with me almost everyday. Could not say the same with the DSLR which rarely see office/street time.

Re:Listen up camera manufacturers

[Jeppe Salvesen]

Agreed. Not everyone wants a huge camera. Not even a over-the-shoulder-sized superzoom or micro four thirds one.

However, interestingly, the point-and-shoots waste a lot of resources and space by exceeding the diffraction limit on common apertures. Plus their "noise-reduction" algorithms is really all about selective downsampling while maintaining file size when you operate within non-limited apertures.

You buy a 12mpx point-and-shoot, but the files themselves are closer to 6-8 mpx in terms of resolution.

Re:Listen up camera manufacturers

[hughk]

I like a larger sensor, but don 't need the extra pixels. It's partly down to the high pixel density why we have so many problems with low-light noise.

Re:Listen up camera manufacturers

[KC7JHO]

No problem, the newer cameras usually have much better optics now, and all you need to do is to set the resolution to a lower setting than the cameras maximum. Most cameras have at least 3 quality settings which only adjust the resolution but still gives you the use of the rest of the improvements over your (presumably) old one. Not saying you should rush out and buy a new camera, just that their is hope when you have to replace the one you now have for some reason. The new camera will also allow you to have the larger format when you need it for that one shot when you just want it to be perfect and to scale to the highest resolution your monitor will handle for a desktop background... etc.

hackable cams already available

[v1]

My PowerShot S3-IS is scriptable. (example) And it's not even a cutting edge camera. Lots of cams support scripting.

Re:hackable cams already available

[mrmojo]

The existence of prior art doesn't mean we shouldn't try and do it better. There's plenty of related work and similar projects.

We're aware of chdk (and have used it for a bunch of stuff), and it's close in some respects, but it's not the same thing. Chdk doesn't turn your camera into a fully functioning linux box, which is part of what we're trying to do, though this has also been done before sans viewfinder (www.elphel.com). You can plug random stuff in over USB, you can control the sensor with extremely low latency (by hacking the kernel if all else fails), you can ssh in, you could even run a web-server off your camera if you wanted to like the elphel cameras. Last week I plugged an SSD drive in over USB (alas no sata interface yet) to save off raw data faster. It's a fairly standard linux so it just worked.

You also have a lot more compute than you might get in something like chdk. You have access to a unified shader architecture GPU, a DSP, a CPU with an SSE-like vector coprocessor, and a fixed-function set of image processing tools (like histogram generation).

The other half of what we're trying to do is make a really good API for a programmable camera, including stuff for synchronization of multiple external devices (eg flashes), optimized image processing routines, frame-level control of the sensor at high frame rates, and camera user interface stuff, including physical widgets like buttons and dials (we use a phidgets board for this).

- Andrew (One of the grad students working on the frankencamera)

Re:hackable cams already available

[mrmojo]

Well sensor noise comes in two main flavors - read noise and photon shot noise. The photon shot noise is inherent in the light - photons come in semi-randomly. The read noise is noise in the sensor circuitry.

If you take 100 pictures with 1/100s exposures and average them, then in terms of photon shot noise it's the same as taking a 1s exposure. You've recorded the same number of photons. However, you paid the read noise penalty 100 times, and while you're still averaging that away, you start with a lot more than if you just took a 1s exposure.

So in the end, your synthetic 1s exposure is noisier than a real 1s exposure, but much less noisy than the longest hand-held exposure you could normally take (~1/60s, depending on focal length).

So it works pretty well.

Could there ever be a view camera version?

[e9th]

This looks promising. But optics being optics. a version with swings & tilts would be really exciting for landscape/architectural/product photographers.

Nikon offer an SDK for many of their cameras

[jamesswift]

http://sdk.nikonimaging.com/apply/

D5000, D3x, D90, D700, D40, D60, D3, D300, D200, D80
And NEF (RAW) files

Re:Do want

[Entropius]

The Canon lens-to-camera communication protocol has been reverse-engineered for a while. Manufacturers like Sigma and Tamron are making (very good) lenses compatible with Canon (and everyone else's) bodies.

Re:Similar work for Canon cameras...

[Entropius]

Is there a way to enable block storage mode on Canon DSLR's?

Seriously, this is a royal PITA. My father is only semi-computer-literate and has a Canon 350D. The hoops he has to jump through in order to get the pictures off of that thing are *insane* -- it literally takes an hour to copy an 8GB CF card, and if he messes up one step in the process he has to start over. (It still takes me forever, but I just let the transfer run in the background).

You could just use a card reader, but the camera craps the pictures into 234897234 different folders -- and he's nowhere near good enough with computers to be able to deal with that.

Meanwhile, I can plug my (also old) Olympus camera into my computer, select "Mass storage" from the menu that appears on the camera, and it's just like a thumbdrive -- with all the pictures in one directory, no funny stuff necessary.

Apparently the modern Canons *still* don't have USB mass storage mode. (Well, not the affordable ones anyway.)

Always wanted a printer

[bogaboga]

I have always wanted an Open Source printer. One that can be built and whose consumables made by those with the means without worrying about patents and all the nonsense. Think about what this could do for students and government departments.

I am personally sick and tired of shelling cash to the Lexmarks, HPs and Epsons of today. Why hasn't this taken off [yet]?

Re:Always wanted a printer

[ledow]

Because a printer costs almost nothing nowadays, certainly less than its component cost to a hobbyist when you can get Lexmark's etc. for about £20 brand new. Old printers are a great source of stepper motors because of this. The major problem is the ink, which is the hard bit to make effectively and cheaply on anything other than a mass scale - so actually with a £20 Lexmark printer and some "clone" ink refill, you've basically got something orders or magnitude more efficient than you could ever make yourself.

The tech being "open source" is a different matter, but we all *know* how printers work. An inkjet is no different to a dot matrix, except instead of hitting a physical pin into an inked ribbon, the carriage heats the ink through a very fine nozzle (probably out of the hobbyists reach without considerable skill/expense). The basic technology is a geared-down belt on a stepper motor. Try getting that working to within 300ths of an inch (300dpi) for less than £20. After that, the actual imprinting technology is either out of reach or difficult to manufacture / time accurately. Then you need control circuitry (WAY more than £20 worth of chip / effort / programming). Then you need the consumable in the first place, which doesn't clog, dry up, leak, etc. And laser printers? Don't even go there for hobbyist use.

Even back in the primarily-hobbyist days, "normal" types of printers around weren't in the hobbyist scene because you couldn't "make" them - the ZX printer on the Spectrum is one example - aluminium-coated paper which was printed on by an electrical "spark" because that's the only way it could be made affordable. Printers are one of those things that you can't make for anywhere near the price that 10 or 12 throwaway ones wouldn't cost. You don't gain anything at all. It's like saying, why doesn't someone make an "open-source" car? Because at the end of the day to make it would actually cost you more than just buying a new car anyway.

Does it really matter?

[gillbates]

Look, here's the disclaimer: I am a software engineer. No, not hardware, software.

But I've written camera drivers from schematics and datasheets alone. It's *just not that hard*. Even for a software guy. I don't have an EE, just an interest in electronics.

And digital electronics are, quite frankly, rather simple. If you know ohm's law, and can read a datasheet or two, you could very easily put together a digital camera module. PCB express will happily etch the board for you, and you *might* have to do some soldering. Unless, of course, you buy one of the cameras from sparkfun or other hobby supplier.

If you can't learn drag and drop PCB design, or can't master basic electronics (ohm's law doesn't even require an understanding of calculus), maybe you shouldn't be tinkering with cameras at the circuit level. A fast fourier transform is far more complicated and difficult to understand than the electronics which go into camera sensors, and yet, is the foundation for all modern video and image compression. If you can't understand that, you will most likely not be contributing much of value to computing applications involving a camera.

Granted, I like open source stuff. But there's already plenty of it out there today - just pick the resolution, frame rate, sensitivity, etc... and go. You don't need the frankencamera. You just need the time and interest, and be willing to spend a few bucks on the hardware.

Re:Does it really matter?

[appoose]

From what I gather, the emphasis for the project is not mainly on facilitating re-implementation of incorporating existing features found in modern cameras, but to enable researchers and developers to add onto to new functionality very seldom found in today's camera. Computational Photography ( http://en.wikipedia.org/wiki/Computational_photography ) is currently a hot research area in academia. A huge chunk of computational photographic work is purely software driven, and do not involve any hardware tweaking (Think HDR imaging, panorama) . Thus, what a programmable (open) camera will enable me is to pick up a recently published work (or device my own technique), implement it and add onto the functionality of the camera. ** snip from the quoted article ** Of course users with Frankencameras would not be constrained by what is already known. Theyâ(TM)d be free to discover and experiment with all kinds of other operations that might yield innovative results because theyâ(TM)d have total control. ** snip ends **

Re:Do want

[mrmojo]

It's not a Nokia imaging chip, it's just the one that happens to be used in Nokia N95s. Aptina makes it and sells it to anyone who wants one. They do make you sign an NDA to get the full data sheet, but that's pretty much impossible to avoid.

As the poster above mentioned, Canon lenses have been thoroughly reverse-engineered.

The lenses would be fairly easy to swap out for a different optical system - we communicate with the lens controller over a simple serial link. The sensor is more involved - for one you'd need a linux kernel driver for your new sensor. Also, it's a pain to properly mount a sensor and get the all support circuitry working. None of it is secret or proprietary though, beyond the NDA you usually need to sign to get the register map for the sensor you want to use.

- Andrew (one of the grad students working on the project)

Why use the EOS mount?

[insane_membrane]

When the best cameras in the world use the F-mount ;)

What about CHDK?

[rdawson]

This kit is FREE open source for the Cannon Powershot, with many of the features mentioned in the article, including HDR. Download it onto a cf flash, and it replaces the Cannon OS. Many amazing images 1/50,000 milkdrop captures, night scenes etc can be found at http://chdk.wikia.com/wiki/CHDK

nice PR stunt

[speedtux]

This sounds like an academic trying to make a name for himself again by labeling something that already exists with his own label. "Computational photography"? Well, how exactly did digital photography ever work without that?

Open source camera OS? Nice try, but the reason manufacturers haven't standardized on anything yet is because the technology keeps changing.

However, FWIW, Canon cameras effectively can be reprogrammed using the CHDK firmware.

Re:nice PR stunt

[mrmojo]

Computational photography is the accepted term for this subfield of computer graphics and computer vision: http://lmgtfy.com/?q=computational+photography

Secondly, we're not making an open source camera OS for existing hardware, we're making camera hardware that runs an existing open source OS - linux - with particular drivers and APIs to help you program the camera.

We're very well aware of CHDK and have used it for many projects. This is not like that (I have an earlier post that elaborates above).

Re:Do want

[mrmojo]

You can find such lens specifications available for most lenses in patent databases. The patents list the curvatures, indices of refraction, etc, etc. That's kind of the point of patents - when used correctly they remove the need for trade secrets. I'm not sure if this holds true for Canon lenses in particular.

I agree with you that many aspects of the hardware are not as open as they could be. However, we're trying to make a camera that actually works well as a camera without reinventing too many wheels. We'd also like to be able to replicate and distribute the camera to other researchers. That means off-the-shelf parts that anyone can buy, like Canon lenses.

- Andrew (one of the grad students involved)

Re:Similar work for Canon cameras...

[Cheesetrap]

You could just use a card reader, but the camera craps the pictures into 234897234 different folders -- and he's nowhere near good enough with computers to be able to deal with that.

One would assume the camera generates unique (or at least very seldom-repeated) filenames for each photo... I don't see what is difficult about a recursive 'move' command? I'm a complete noob @ programming but it only takes a one-line batch file..

<20 minutes later>

Okay, damn you for making me work it out, here you go :P

File: dumpdir.bat

@echo off
set listfile=c:\temp\list.txt
if "%1"=="." (echo %cd%>%listfile%) else (echo %1>%listfile%)
dir %1 /b/s/a:d>>%listfile%
for /f "tokens=*" %%s in (%listfile%) do (xcopy "%%s"%4 %2 %3 /v /f /z)

And this can be called with:

Normal usage:dumpdir path1 path2

Test mode (lists files only): dumpdir path1 path2 /L

Restrict to matching string: dumpdir path1 path2 /R \*.ext (backslash required)

Note that path1 and path2 must be enclosed in quotes if they include spaces - and if you are running the batch from the directory you are wanting to consolidate, you can use a period instead of typing out the path (e.g, dumpdir . c:\pics).

Hope that helps someone ;)

Where's the source?

[ITMagic]

Ermm, Just where, exactly, is the source? Links to CAD designs, specs, software, etc? As far as following the Stanford links is concerned, I can't see how this project can be labeled "Open Source" without this...

Re:Open standards

[phozz bare]

If you're out, I have a few extras you can borrow:

s s s s
  s s s

You're welcome!