A RAW repository, The Internet Archive and OpenRAW

Stan writes "I just read this in the OpenRAW mailing list, OpenRAW plans to create a RAW repository, a final resting place for RAW file documentations of current and already abandoned digital cameras. The RAW repository will be hosted in the Internet Archive, which describes themselves as a digital archive of the Internet and other cultural artifacts. And they have all reasons to support OpenRAW, they currently photograph billions of book pages with cameras and store them in RAW format. Unfortunately the camera makers think different (which is not always a good thing)."

Comments

What's been said before

[twoshortplanks]

In the last slashdot article on OpenRAW

Re:why?

[BabyDave]

I think it refers to the camera's internal representation of the image (i.e. the "raw" data)

So does basically every camera

[Jonathan]

The point isn't that you can get a jpg out of your camera -- I haven't seen a digital camera that can't -- the problem is that the original, uncompressed data generally isn't in an open format.

going canon fanboy in here but

[william_w_bush]

so far my experience with canon gear has only been positive. firstly, they released linux and osx drivers for an old inkjet i had, along with most of their current line, and a good number of their scanners. second, their cameras are amazing, and use normal sd and cf cards instead of the MS and XD that are becoming infuriatingly ubiquitous. Also, their printer lines tend to standardize on the same types of ink, with better quality than the hp's and terrible machines epson is putting out nowadays (my r300 photo printer ran low on lt yellow ink, so it won't print black and white and keeps nagging me in windows to order more ink).

i suspect this is just canon usa marketing dicks playing bs politics for their own sake. so far theyve given out a lot better specs for most of their printers than most companies, and few printer mfg's will even bother to put out cups drivers for their lines.

not releasing their RAW format seems amazingly petty, but sounds exactly like all those fat, middle-aged sales execs who thought it wasn't worth it developing open-sourced linux drivers, cause they could get more commision charging each customer for the drivers themselves. we released them anyway, but a lot of those types make VP and do stupid shit like this to try to throw their cock around.

Re:why?

[petermgreen]

in this case (cameras) raw reffers to the fact its the raw unprocessed data from the CCD

this has to be processed to convert it to a form that we would recognise as an image file. This can happen either on the camera or on a PC.

However This conversion process may well not be fully reversable (due to rounding errors) and bloats the data considerablly (CCDs generally make a red green OR blue value at each location image files generally have red green AND blue at each location so turning CCD output into an image file always involves interpolation) so from an archivists point of view its best to keep the raw data unfortunately that raw data is often in a closed format.

Just keep...

[troon]

...a copy of the dcraw source code.

Re:Just keep...

[RDW]

The various manufacturer-specific Perl modules in the ExifTool package are also an excellent source of documentation for RAW file metadata. Reading this (rather well-commented) code can help make the more cryptic dcraw source much more comprehensible.

Re:What horseshit

[Hast]

Unfortunately DNG doesn't go all the way and just moves the problem further down the line. From the OpenRAW FAQ:

DNG also allows "private data" to be stored in the DNG file. This private data is only known to the camera company that wrote the private data. Third party software that reads and/or writes DNG files will ignore private data recorded by the camera. Only the software written by the camera maker will read the private data written to the DNG file by its camera. Some of this private data might be important or useful information needed by a RAW converter. Adobe's DNG format does not eliminate the problem of undocumented RAW files but transfers the problem into another "container", the DNG file. By allowing private (undocumented) data in the DNG file, DNG does not meet OpenRAW's goals.

So in the end you end up with a similar situation. You can read the data but you don't know exactly how to treat it.

Re:Digital == Loss of freedom

[aaronl]

Actually... for the most part the formulas are publicly known for developers. You don't have to worry about acid fixes and rinse baths being proprietary. For most developers there are a number of companies that will make the chemical for you. Then you can just mix it yourself, or dilute it. I suppose you could make it yourself from scratch, but many of them use all sorts of evil to arrive at the final chemistry.

Also, it isn't particuarly impossible to make your own film. Sure, getting the emulsion nice-nice isn't easy, but it's possible to do yourself.

All that aside, you can develop most negatives from any vendor in the same chemistry. It's the C-41 process, and it makes industries like one-hour photofinishing possible. It also makes your life easier as an enthusiast.

IOW, you don't have to worry about film being shut down because Kodak or whoever doesn't feel like manufacturing any more. It might be more difficult to get your favorite chems is all.

you need to know what you are doing

[cahiha]

By converting, for example, to JPEG and throwing the RAW away you are losing lots of creative post-processing control.

Indeed, you are. That's because JPEG doesn't have the depth and gamut to represent digital images captured by modern cameras. If you use a more modern cooked, open format (e.g., TIFF, JPEG2000) at the right depth and at a lossless or high quality setting, then you lose nothing.

Later you may want to re-process the picture with superior software (it does get better over time).

And you can do that because the interpolation from raw to RGB is invertible given a tiny bit of metadata (metadata you would need to carry around with the raw image as well). If you are dealing with a four channel sensor, you can still store the data in a four channel TIFF. No need to store RAW.

Or perhaps you just needed to tweak the white balance.

You can do that in RGB.

Or fiddle with the sharpening.

For professional archival applications, you should turn off in-camera sharpening. That has nothing to do with whether you use raw storage.

Or the tone curve.

You can do that on the cooked image just as easily as on the raw.

I think the whole confusion arises because people like you equate non-raw storage with JPEG. JPEG is a bad choice for high-quality archival storage, but formats like TIFF and JPEG2000 are fine.

Summary of some key points:

[Distan]

What is this "RAW" format?

RAW doesn't really refer to any single file format. RAW refers to pulling the unprocessed (raw) sensor data out of a digital camera. The actual layout of the bits varies from brand to brand, and often from model to model.

Why do photographers want access to the raw data anyway?

Many professional/prosumer photographers like to archive the version of their work that contains as much of the originally captured information as possible. In the professional film world, this meant processed slides (for consumers, this meant processed negatives). In the digital world, the RAW file contains all the data captured by the camera, before some data is lost by compression and other data is added through interpolation.

Can't they just pull a lossless image out of the camera and be happy?

No. The very act of converting the raw data into an image involves lossful processing of the data. Out of gamut color data is discarded, and CCD color data is interpolated to fill surrounding pixels.