Well sensor noise comes in two main flavors - read noise and photon shot noise. The photon shot noise is inherent in the light - photons come in semi-randomly. The read noise is noise in the sensor circuitry.
If you take 100 pictures with 1/100s exposures and average them, then in terms of photon shot noise it's the same as taking a 1s exposure. You've recorded the same number of photons. However, you paid the read noise penalty 100 times, and while you're still averaging that away, you start with a lot more than if you just took a 1s exposure.
So in the end, your synthetic 1s exposure is noisier than a real 1s exposure, but much less noisy than the longest hand-held exposure you could normally take (~1/60s, depending on focal length).
Well no, to be honest. Right now we're using Canon lenses in front of a tiny sensor. This gives us a 6x crop factor, so we're wasting almost all of the field of view of the lens. We're currently working on replacing the sensor with something larger.
Yeah, that's exactly the kind of thing we're talking about. You need low level access to the sensor for it to work. In fact, that idea works so well that its one of the few things from the area that has made it into a commercial camera - the Casio EX-F1 does it.
You can find such lens specifications available for most lenses in patent databases. The patents list the curvatures, indices of refraction, etc, etc. That's kind of the point of patents - when used correctly they remove the need for trade secrets. I'm not sure if this holds true for Canon lenses in particular.
I agree with you that many aspects of the hardware are not as open as they could be. However, we're trying to make a camera that actually works well as a camera without reinventing too many wheels. We'd also like to be able to replicate and distribute the camera to other researchers. That means off-the-shelf parts that anyone can buy, like Canon lenses.
Computational photography is the accepted term for this subfield of computer graphics and computer vision: http://lmgtfy.com/?q=computational+photography
Secondly, we're not making an open source camera OS for existing hardware, we're making camera hardware that runs an existing open source OS - linux - with particular drivers and APIs to help you program the camera.
We're very well aware of CHDK and have used it for many projects. This is not like that (I have an earlier post that elaborates above).
It's not a Nokia imaging chip, it's just the one that happens to be used in Nokia N95s. Aptina makes it and sells it to anyone who wants one. They do make you sign an NDA to get the full data sheet, but that's pretty much impossible to avoid.
As the poster above mentioned, Canon lenses have been thoroughly reverse-engineered.
The lenses would be fairly easy to swap out for a different optical system - we communicate with the lens controller over a simple serial link. The sensor is more involved - for one you'd need a linux kernel driver for your new sensor. Also, it's a pain to properly mount a sensor and get the all support circuitry working. None of it is secret or proprietary though, beyond the NDA you usually need to sign to get the register map for the sensor you want to use.
- Andrew (one of the grad students working on the project)
I hear you - we want the same thing. Our target is basically a Nokia N900 (which covers 1, 2, 3, 4 and runs linux to boot), plus a much higher quality sensor and lens.
- Andrew (one of the grad students working on the project).
The existence of prior art doesn't mean we shouldn't try and do it better. There's plenty of related work and similar projects.
We're aware of chdk (and have used it for a bunch of stuff), and it's close in some respects, but it's not the same thing. Chdk doesn't turn your camera into a fully functioning linux box, which is part of what we're trying to do, though this has also been done before sans viewfinder (www.elphel.com). You can plug random stuff in over USB, you can control the sensor with extremely low latency (by hacking the kernel if all else fails), you can ssh in, you could even run a web-server off your camera if you wanted to like the elphel cameras. Last week I plugged an SSD drive in over USB (alas no sata interface yet) to save off raw data faster. It's a fairly standard linux so it just worked.
You also have a lot more compute than you might get in something like chdk. You have access to a unified shader architecture GPU, a DSP, a CPU with an SSE-like vector coprocessor, and a fixed-function set of image processing tools (like histogram generation).
The other half of what we're trying to do is make a really good API for a programmable camera, including stuff for synchronization of multiple external devices (eg flashes), optimized image processing routines, frame-level control of the sensor at high frame rates, and camera user interface stuff, including physical widgets like buttons and dials (we use a phidgets board for this).
- Andrew (One of the grad students working on the frankencamera)
That's exactly right - that way it's free for poor people. As unpopular as it is in the US, redistribution of wealth is a hallmark of civilization worldwide.
I'm Palo Alto, in the heart of silicon valley. The GPS fix time on mine is often five minutes even when the menu shows many satellites visible with good signal strength.
I'm interning at Nokia Research labs, and they give us all N95s - top of the line do everything phones. I still miss my Nokia 1100 though. It's the only phone I've ever had with buttons that are actually comfortable to press. I don't feel like I need a pair of tweezers to operate it. The battery lasted forever too, I guess that's the benefit of having no features:)
I'm sure that even if you take longer than 48 hours to prove the DMCA notice is incorrect, they won't charge you, and will reconnect you. Stanford admins tend to be reasonable human beings.
The reasons Stanford did this are very simple, are explained in detail by Stanford, and were entirely left out of the summary. They have to pay 3 people full time to respond to DMCA complaints, the vast majority of which are caused by students doing something wrong. The salary these three people earn would be better spent on education, and it isn't fair that the entire student body has to pay for the acts of the ones infringing. They're shifting the costs of responding to these DMCA complaints to the students who cause them.
As a show of good faith, for the first year all collected money will not offset the salaries, but will in fact go directly to the student government.
To reiterate, they're just shifting the costs of responding to DMCA complaints onto the students too dumb to get their warez and MP3s from usenet like the rest of us;)
Heavens no, we don't want experts in the law making new laws. Before we know it people will be hiring software engineers to write code and architects to design buildings!
Why is this funny? When I was a section leader for undergraduate computer science, I forbid note-taking of any kind in my sections. It made for a much more interactive class. Everyone had more fun, and everyone learnt more.
Think of it this way, adding up the 12x12 pixels under a microlens is exactly the same as having a single larger sensor there, in terms of the physical process occuring. Sensor noise per pixel is not significant compared to photon noise.
Therefore, because your sqrt(N) argument is correct, partitioning the pixel should only increase noise by a factor of 12, not 144.
I'm one of the guys involved in this research at Stanford. I just posted below explaining that due to noise issues, you don't really fundamentally have to lose resolution, as you can just use an equally sized sensor with cell phone camera size pixels instead of good camera size pixels, and when you add up pixels to create a focused image, the noise goes down to match noise levels in a good camera.
I'm one of the guys who works on this stuff at Stanford. I should point out that it's not fair to say you lose resolution, because good cameras have large pixels to reduce noise over a finite exposure time. Lightfield cameras, because they add up a whole lot of individual pixel samples to produce an image pixel, can get away with much much smaller pixels, because the noise goes down as you sum up the pixel values.
The best way to think of it is take a standard good quality camera with big pixels, subdivide each pixel into a grid of 12x12 or so tiny pixels - more like the size of pixels in cell phone cameras - and put a microlens over it. You get the same spatial resolution as the good camera, roughly the same noise characteristics, and the ability to refocus and pull other light field tricks like hitchcock zooms.
You just have to be aware that treating the data as a light field it's very noisy, like a crappy cell phone camera, but when you add up pixels to make a focused image, the noise drops back to regular good camera levels.
It's just harder to deal with the amount of data you get off a large sensor with tiny pixels, and they're also harder to build, but neither point is a showstopper and these are mere engineering issues...
Even with one eye you see the effect. Leonardo da Vinci noted that if you hold a pin close to one eye, it disappears because it's significantly smaller than your pupil.
They are separate, but the symphonies are even better. Try listening to the last movement of the 9th and the second movement of the 7th in particular. Those are very accessible.
Slashdot Mirror
Well sensor noise comes in two main flavors - read noise and photon shot noise. The photon shot noise is inherent in the light - photons come in semi-randomly. The read noise is noise in the sensor circuitry.
If you take 100 pictures with 1/100s exposures and average them, then in terms of photon shot noise it's the same as taking a 1s exposure. You've recorded the same number of photons. However, you paid the read noise penalty 100 times, and while you're still averaging that away, you start with a lot more than if you just took a 1s exposure.
So in the end, your synthetic 1s exposure is noisier than a real 1s exposure, but much less noisy than the longest hand-held exposure you could normally take (~1/60s, depending on focal length).
So it works pretty well.
Well no, to be honest. Right now we're using Canon lenses in front of a tiny sensor. This gives us a 6x crop factor, so we're wasting almost all of the field of view of the lens. We're currently working on replacing the sensor with something larger.
Yeah, that's exactly the kind of thing we're talking about. You need low level access to the sensor for it to work. In fact, that idea works so well that its one of the few things from the area that has made it into a commercial camera - the Casio EX-F1 does it.
You can find such lens specifications available for most lenses in patent databases. The patents list the curvatures, indices of refraction, etc, etc. That's kind of the point of patents - when used correctly they remove the need for trade secrets. I'm not sure if this holds true for Canon lenses in particular.
I agree with you that many aspects of the hardware are not as open as they could be. However, we're trying to make a camera that actually works well as a camera without reinventing too many wheels. We'd also like to be able to replicate and distribute the camera to other researchers. That means off-the-shelf parts that anyone can buy, like Canon lenses.
- Andrew (one of the grad students involved)
Computational photography is the accepted term for this subfield of computer graphics and computer vision: http://lmgtfy.com/?q=computational+photography
Secondly, we're not making an open source camera OS for existing hardware, we're making camera hardware that runs an existing open source OS - linux - with particular drivers and APIs to help you program the camera.
We're very well aware of CHDK and have used it for many projects. This is not like that (I have an earlier post that elaborates above).
Because we have a lot of Canon lenses lying around. It's not a fundamental part of the system - it would be easy to switch in a different lens mount.
- Andrew (one of the grad students involved)
It's not a Nokia imaging chip, it's just the one that happens to be used in Nokia N95s. Aptina makes it and sells it to anyone who wants one. They do make you sign an NDA to get the full data sheet, but that's pretty much impossible to avoid.
As the poster above mentioned, Canon lenses have been thoroughly reverse-engineered.
The lenses would be fairly easy to swap out for a different optical system - we communicate with the lens controller over a simple serial link. The sensor is more involved - for one you'd need a linux kernel driver for your new sensor. Also, it's a pain to properly mount a sensor and get the all support circuitry working. None of it is secret or proprietary though, beyond the NDA you usually need to sign to get the register map for the sensor you want to use.
- Andrew (one of the grad students working on the project)
I hear you - we want the same thing. Our target is basically a Nokia N900 (which covers 1, 2, 3, 4 and runs linux to boot), plus a much higher quality sensor and lens.
- Andrew (one of the grad students working on the project).
The existence of prior art doesn't mean we shouldn't try and do it better. There's plenty of related work and similar projects.
We're aware of chdk (and have used it for a bunch of stuff), and it's close in some respects, but it's not the same thing. Chdk doesn't turn your camera into a fully functioning linux box, which is part of what we're trying to do, though this has also been done before sans viewfinder (www.elphel.com). You can plug random stuff in over USB, you can control the sensor with extremely low latency (by hacking the kernel if all else fails), you can ssh in, you could even run a web-server off your camera if you wanted to like the elphel cameras. Last week I plugged an SSD drive in over USB (alas no sata interface yet) to save off raw data faster. It's a fairly standard linux so it just worked.
You also have a lot more compute than you might get in something like chdk. You have access to a unified shader architecture GPU, a DSP, a CPU with an SSE-like vector coprocessor, and a fixed-function set of image processing tools (like histogram generation).
The other half of what we're trying to do is make a really good API for a programmable camera, including stuff for synchronization of multiple external devices (eg flashes), optimized image processing routines, frame-level control of the sensor at high frame rates, and camera user interface stuff, including physical widgets like buttons and dials (we use a phidgets board for this).
- Andrew (One of the grad students working on the frankencamera)
That's exactly right - that way it's free for poor people. As unpopular as it is in the US, redistribution of wealth is a hallmark of civilization worldwide.
Nice post. I was thinking something along those lines, but I had no chance of putting into words as eloquently as you have.
I'm Palo Alto, in the heart of silicon valley. The GPS fix time on mine is often five minutes even when the menu shows many satellites visible with good signal strength.
I'm interning at Nokia Research labs, and they give us all N95s - top of the line do everything phones. I still miss my Nokia 1100 though. It's the only phone I've ever had with buttons that are actually comfortable to press. I don't feel like I need a pair of tweezers to operate it. The battery lasted forever too, I guess that's the benefit of having no features :)
I'm sure that even if you take longer than 48 hours to prove the DMCA notice is incorrect, they won't charge you, and will reconnect you. Stanford admins tend to be reasonable human beings.
The reasons Stanford did this are very simple, are explained in detail by Stanford, and were entirely left out of the summary. They have to pay 3 people full time to respond to DMCA complaints, the vast majority of which are caused by students doing something wrong. The salary these three people earn would be better spent on education, and it isn't fair that the entire student body has to pay for the acts of the ones infringing. They're shifting the costs of responding to these DMCA complaints to the students who cause them.
;)
As a show of good faith, for the first year all collected money will not offset the salaries, but will in fact go directly to the student government.
To reiterate, they're just shifting the costs of responding to DMCA complaints onto the students too dumb to get their warez and MP3s from usenet like the rest of us
Heavens no, we don't want experts in the law making new laws. Before we know it people will be hiring software engineers to write code and architects to design buildings!
Are you referring to 'learnt'?
It's the British english form of learned. I'm from Australia.
Now that I'm living in the US, my spelling has become a horrible hybrid of the two conventions.
Why is this funny? When I was a section leader for undergraduate computer science, I forbid note-taking of any kind in my sections. It made for a much more interactive class. Everyone had more fun, and everyone learnt more.
Therefore, because your sqrt(N) argument is correct, partitioning the pixel should only increase noise by a factor of 12, not 144.
Good point, and an excellent question. We're looking into it.
I'm one of the guys involved in this research at Stanford. I just posted below explaining that due to noise issues, you don't really fundamentally have to lose resolution, as you can just use an equally sized sensor with cell phone camera size pixels instead of good camera size pixels, and when you add up pixels to create a focused image, the noise goes down to match noise levels in a good camera.
The best way to think of it is take a standard good quality camera with big pixels, subdivide each pixel into a grid of 12x12 or so tiny pixels - more like the size of pixels in cell phone cameras - and put a microlens over it. You get the same spatial resolution as the good camera, roughly the same noise characteristics, and the ability to refocus and pull other light field tricks like hitchcock zooms.
You just have to be aware that treating the data as a light field it's very noisy, like a crappy cell phone camera, but when you add up pixels to make a focused image, the noise drops back to regular good camera levels.
It's just harder to deal with the amount of data you get off a large sensor with tiny pixels, and they're also harder to build, but neither point is a showstopper and these are mere engineering issues...
Even with one eye you see the effect. Leonardo da Vinci noted that if you hold a pin close to one eye, it disappears because it's significantly smaller than your pupil.
If only I could used my mod points to mod you up further than +5 funny.
They are separate, but the symphonies are even better. Try listening to the last movement of the 9th and the second movement of the 7th in particular. Those are very accessible.