A RAW repository, The Internet Archive and OpenRAW

Stan writes "I just read this in the OpenRAW mailing list, OpenRAW plans to create a RAW repository, a final resting place for RAW file documentations of current and already abandoned digital cameras. The RAW repository will be hosted in the Internet Archive, which describes themselves as a digital archive of the Internet and other cultural artifacts. And they have all reasons to support OpenRAW, they currently photograph billions of book pages with cameras and store them in RAW format. Unfortunately the camera makers think different (which is not always a good thing)."

Comments

Which format again...?

[Anonymous Coward]

I just read this in the OpenRAW mailing list, OpenRAW plans to create a RAW repository, a final resting place for RAW file documentations of current and already abandoned digital cameras. The RAW repository will be hosted in the Internet Archive

Sorry, you lost me there. Which format will this archive be covering?

Digital == Loss of freedom

[Dancin_Santa]

I'm not going to say I told you so, but I told you so. The minute you give up the physical artifact and rely on a digital representation of your data, you are at the risk of any company who wishes to exert some control over the format of that data. That's why all those RAW file formats for each camera are different from company to company. They gain the most benefit by locking you into a certain piece of software and forcing you along their upgrade path.

If you stick with film, you are only limited in your ability to develop your own negatives. If you can do this, you will be able to continue with film for as long as you want. Scan the negs and save them in whatever format you want. It doesn't matter because the actual physical artifact is still in your possession.

Not so with Digital.

In many ways, digital is superior to film. However, when it comes to ownership of your data, you are far better off with film than you ever can be with digital.

Re:Digital == Loss of freedom

[kimba]

With my camera I shoot in RAW. By some process in history, today the RAW format for my specific camera is open - available not only for use in commercial products like Adobe, but in GPL'd software that will convert it for me and for while I have the source.

Unless someone arrests me and confiscates all my software, as well as removes all this purportedly legal software from the market, what is the risk of using this camera?

Re:Digital == Loss of freedom

[phidipides]

They gain the most benefit by locking you into a certain piece of software and forcing you along their upgrade path.

Just to nitpick a bit, most of the professional photographers I know use the various plugins to Photoshop to work with RAW images, so technically they aren't being forced into an upgrade path by the camera manufacturers. I personally use a Canon 10D, and the Canon software is so awful that I always use other tools to convert and manipulate the images.

I fully agree with your point that it would be better if the camera manufacturers fully opened up their file formats, and I fail to see how keeping them closed provides them an actual competetive advantage. However, so long as there is no constraint against converting RAW images to another lossless format I'm not sure that this is a battle in which the camera companies can be accused of trying to pull a fast one on consumers; I think it's merely a case where they need to be educated about the further benefits of opening up their formats (ie open source developers can build free tools, etc).

Re:Digital == Loss of freedom

[Hast]

Oh you mean you'd rather use propriatary film with propriatary developer chemicals? There is nothing particularly "open" about that besides that the chemicals are available to buy in most photography stores. If they go out you will no longer be able to develop your old exposed film.

Just to be clear, RAW is like the undeveloped exposed negative. After "developing" it to a TIFF16 or whatever format you want to have. You might think that RAW is equivalent to the undeveloped negative, but it really isn't.

Besides, there is always DCRAW which allows you to "develop" your RAW files in an OSS fasion.

Furthermore the reason RAW formats vary between makers is because it is raw data from the CCD/CMOS. So it's not strange at all that different manufacturers use different formats.

I do agree with you though that we need open standards as far as RAW is concerned. I don't agree that the film world is any better though.

Re:Digital == Loss of freedom

[smithberry]

There is always some risk in machine readable data.
In 20 or 30 years time you might have trouble getting hardware which reads your data and runs an OS which runs your software.
Ok, you might be clued up enough to always copy backups to newer technology, but joe public is one day going to bring a CD out of his dad's attic and find he cannot even look at the photos on it.

Re:Digital == Loss of freedom

[Phanatic1a]

There is always some risk in machine readable data.

There is always some risk, period. If I keep stacks of negatives in boxes in my house, then in 20 or 30 years time my house might catch fire and burn to the ground.

Okay, *I* might be clued up enough to always keep my negatives in a fireproof safe, but Joe Public is one day going to use a penny as a fuse replacement and find that he can't look at the photos on all that celluloid ash.

What's been said before

[twoshortplanks]

In the last slashdot article on OpenRAW

RAW format

[James_Duncan8181]

The RAW format wars seem an odd competiton between camera manufacturers, who are actively hurting their presence in the professional space by making their imafes less useful for archive purposes and less interoperable for press agencies to sell. The thing that is particually noticible is that manufacturers are now being actively co-opted into sharing this information under NDA with MS to allow the hardware to work seamlessly with Longhorn. This mass move from open to propriatory standards (something MS will, of course, encourage) is meaning that the camera manufacturers are seeing their poduct become commoditised, and apparently feeling unable to compete on hardware quality alone.

Thanks Canon, you just made me finally feel confident about buying Taiwanese.

Re:RAW format

[DigicamGuy]

The thing that is particually noticible is that manufacturers are now being actively co-opted into sharing this information under NDA with MS to allow the hardware to work seamlessly with Longhorn.

Actually, that's incorrect. The Longhorn interface is binary-only (no source code or format information is communicated to Microsoft or to the OS). Basically, the manufacturer (or third-party developer) writes a driver with an API that makes processed RGB data available to the OS. This is the same basic mode of operation as Canon and Nikon (and probably others) have implemented already in their free SDKs. Here's a brief interview with a Microsoft exec about the Longhorn interface and the shortly forthcoming "powertoy" RAW thumbnailer/viewer that's coming for XP. -- Not likely the level of detail /. people would want, but more than I've seen elsewhere, may help dispel some of the misconceptions.

Of course, this means that the proprietary RAW formats remain entirely proprietary in the Longhorn era.

For the record, I personally think that some level of open documentation of RAW formats makes a whole lot more sense than trying to come up with a common standard. A number of people (Adobe prominent among them, of course) have proposed Adobe DNG as a "universal" format. This sounds like a wonderful idea until you look at the assumptions underlying the format: It assumes a rectilinear pixel array, with a Bayer color filter array pattern (a checkerboard of RGB color filters on the pixels, with twice as many green pixels as red or blue). This is indeed the format used by the majority of cameras out there, but it completely misses innovations such as Foveon's full-RGB-in-every-pixel sensor, Fuji's hexagonal-pixel/diagonal-array "SuperCCD", and Fuji's latest "SR" sensors, which combine low- and high-sensitivity sensors in each pixel.

While a "universal" RAW format would help with the issue of access to the underlying data, so would simple documentation of the structure of various proprietary RAW formats, and the latter wouldn't have the negative effect of stifling innovation in sensor technology.

Re:Can anyone say...

[meringuoid]

Somewhere in the world where the Americans aren't in control? Closest you're going to get is probably Iraq...

How Open is the Repository?

[Anonymous Coward]

This effort is being set up by a guy (Juergen Specht) who hosted a mailing list and then deleted it without notice when some of the posts offended him.

See:
http://www.vudeja.com/04/09/mailing-list

http://www.esthet.org/blog/archives/001294.html

http://www.wirefarm.com/archives/004186.html

http://www.easterwood.org/hmmn/archives/001111.htm l

http://openraw.org/about/

Don't be surprised if this site just up and disappears one day, taking all of the data with it.

What horseshit

[loraksus]

The manufacturers are just opposed to working together to create some sort of standard.

But can you blame them? Really, think about this for a second - people (scumbag fucks who should hang from lampposts, call them what you will) from Rambus sat in standards groups for years and then turned around and secretly patented the standard and then had the balls to demand royalties. You saw more or less the same bullshit with .gif and in hundreds of other similar cases in the last 20 or so years.
I think it is (sort of) understandable that companies would be hesistant to work together to develop a standard way of doing something - especially in a cutthroat business such as photography.

And by the way, using Canon is a fairly shitty example, Nikon is far worse when it comes to the RAW format (ok, its not really a format) bullshit that flows through the world of pro photography.

That all said, this smacks more of the petty bickering that is involved in cameras more than than anything else (See Also, "Complete lack of lens interchangability" et al), but as always, we (or those who buy $600+ cameras) get fucked.

Don't get me started on how "using the DMCA to "protect" the super complex almost but not quite encrypted raw format". I don't need a stroke at this age. . .

So does basically every camera

[Jonathan]

The point isn't that you can get a jpg out of your camera -- I haven't seen a digital camera that can't -- the problem is that the original, uncompressed data generally isn't in an open format.

Need for a broader approach?

[archeopterix]

Hm... Everytime I read something similar to the article (that is about twice a week), I think that there should be a non-profit organisation to oppose taking away customer rights under the guise of "intellectual property".

Something powerful enough to organise boycott that would cause *pain* to the offending company. Something that a congresscritter would be afraid to piss off. EFF comes close, except that it a) has a broader scope and b) sadly is not powerful enough.

Too bad that the existing consumer organisations are focused on making money from their "consumer reports" and the general population doesn't care (the frog is half-boiled and still comfortable).

Re:why?

[petermgreen]

in this case (cameras) raw reffers to the fact its the raw unprocessed data from the CCD

this has to be processed to convert it to a form that we would recognise as an image file. This can happen either on the camera or on a PC.

However This conversion process may well not be fully reversable (due to rounding errors) and bloats the data considerablly (CCDs generally make a red green OR blue value at each location image files generally have red green AND blue at each location so turning CCD output into an image file always involves interpolation) so from an archivists point of view its best to keep the raw data unfortunately that raw data is often in a closed format.

Re:storing raw digital images is stupid

[jgordon7]

Most raw formats on modern cameras is compressed using a lossless format. Take Nikons nef format, a 6.1mp image in raw is only about 4-5 MB, if this was not compressed it would be closer to 10 MB.

You can not tweak certain settings as easily once the image is convert to another format, even a lossless one like tiff. Best example is white balance.

Re:Please tell me this, this is critical..

[Ubergrendle]

From Canon, as they refused to cooperate with openRAW and ended their letter with a slap in the face: "If our equipment or software does not meet your needs, you are entirely welcome to seek other suppliers".

This is true, and unfortunately Canon can afford to take this position. In the DSLR market -- the *serious* digital photography market -- Canon has through various reports a 50-70% market share. Their only serious competitor is Nikon who controls anywhere from 30-50% depending upon who you listen to, and the rest make up a very small percentage. Kodak just announced a complete retirement from the DSLR market, Sigma cameras are doing horribly, and although Pentax and Minolta have decent offerings their market penetration is relatively weak. Canon can throw around threats since Nikon is WORSE in their disregard for RAW, actively encrypting (weakly) the white balance data. Nikon knows the encryption is a joke, but its enough to have legal teeth via the DCMA and thus Adobe won't translate it.

Personally I'm more concerned with the retirement of RAW formats than the current vendor specificity. When you by a Canon EOS system or a Nikon F-mount you're buying into a closed, proprietary hardware system. Extending it to the software realm is crappy, but not surprising. Microsoft is best positioned to bust this wide open, and its in Adobe's best interests to open RAW or see the success of DNG. My guess is once the balance of power starts shifting heavily in favour of Canon or Nikon (towards virtual monopoly) the lesser company will open up their RAW format to be more accomodating.

Summary of some key points:

[Distan]

What is this "RAW" format?

RAW doesn't really refer to any single file format. RAW refers to pulling the unprocessed (raw) sensor data out of a digital camera. The actual layout of the bits varies from brand to brand, and often from model to model.

Why do photographers want access to the raw data anyway?

Many professional/prosumer photographers like to archive the version of their work that contains as much of the originally captured information as possible. In the professional film world, this meant processed slides (for consumers, this meant processed negatives). In the digital world, the RAW file contains all the data captured by the camera, before some data is lost by compression and other data is added through interpolation.

Can't they just pull a lossless image out of the camera and be happy?

No. The very act of converting the raw data into an image involves lossful processing of the data. Out of gamut color data is discarded, and CCD color data is interpolated to fill surrounding pixels.