Slashdot Mirror


A RAW repository, The Internet Archive and OpenRAW

Stan writes "I just read this in the OpenRAW mailing list, OpenRAW plans to create a RAW repository, a final resting place for RAW file documentations of current and already abandoned digital cameras. The RAW repository will be hosted in the Internet Archive, which describes themselves as a digital archive of the Internet and other cultural artifacts. And they have all reasons to support OpenRAW, they currently photograph billions of book pages with cameras and store them in RAW format. Unfortunately the camera makers think different (which is not always a good thing)."

36 of 146 comments (clear)

  1. Which format again...? by Anonymous Coward · · Score: 4, Funny
    I just read this in the OpenRAW mailing list, OpenRAW plans to create a RAW repository, a final resting place for RAW file documentations of current and already abandoned digital cameras. The RAW repository will be hosted in the Internet Archive
    Sorry, you lost me there. Which format will this archive be covering?
  2. Digital == Loss of freedom by Dancin_Santa · · Score: 5, Insightful

    I'm not going to say I told you so, but I told you so. The minute you give up the physical artifact and rely on a digital representation of your data, you are at the risk of any company who wishes to exert some control over the format of that data. That's why all those RAW file formats for each camera are different from company to company. They gain the most benefit by locking you into a certain piece of software and forcing you along their upgrade path.

    If you stick with film, you are only limited in your ability to develop your own negatives. If you can do this, you will be able to continue with film for as long as you want. Scan the negs and save them in whatever format you want. It doesn't matter because the actual physical artifact is still in your possession.

    Not so with Digital.

    In many ways, digital is superior to film. However, when it comes to ownership of your data, you are far better off with film than you ever can be with digital.

    1. Re:Digital == Loss of freedom by kimba · · Score: 4, Interesting

      With my camera I shoot in RAW. By some process in history, today the RAW format for my specific camera is open - available not only for use in commercial products like Adobe, but in GPL'd software that will convert it for me and for while I have the source.

      Unless someone arrests me and confiscates all my software, as well as removes all this purportedly legal software from the market, what is the risk of using this camera?

    2. Re:Digital == Loss of freedom by phidipides · · Score: 5, Insightful

      They gain the most benefit by locking you into a certain piece of software and forcing you along their upgrade path.

      Just to nitpick a bit, most of the professional photographers I know use the various plugins to Photoshop to work with RAW images, so technically they aren't being forced into an upgrade path by the camera manufacturers. I personally use a Canon 10D, and the Canon software is so awful that I always use other tools to convert and manipulate the images.

      I fully agree with your point that it would be better if the camera manufacturers fully opened up their file formats, and I fail to see how keeping them closed provides them an actual competetive advantage. However, so long as there is no constraint against converting RAW images to another lossless format I'm not sure that this is a battle in which the camera companies can be accused of trying to pull a fast one on consumers; I think it's merely a case where they need to be educated about the further benefits of opening up their formats (ie open source developers can build free tools, etc).

    3. Re:Digital == Loss of freedom by Hast · · Score: 3, Insightful

      Oh you mean you'd rather use propriatary film with propriatary developer chemicals? There is nothing particularly "open" about that besides that the chemicals are available to buy in most photography stores. If they go out you will no longer be able to develop your old exposed film.

      Just to be clear, RAW is like the undeveloped exposed negative. After "developing" it to a TIFF16 or whatever format you want to have. You might think that RAW is equivalent to the undeveloped negative, but it really isn't.

      Besides, there is always DCRAW which allows you to "develop" your RAW files in an OSS fasion.

      Furthermore the reason RAW formats vary between makers is because it is raw data from the CCD/CMOS. So it's not strange at all that different manufacturers use different formats.

      I do agree with you though that we need open standards as far as RAW is concerned. I don't agree that the film world is any better though.

    4. Re:Digital == Loss of freedom by smithberry · · Score: 3, Insightful

      There is always some risk in machine readable data.
      In 20 or 30 years time you might have trouble getting hardware which reads your data and runs an OS which runs your software.
      Ok, you might be clued up enough to always copy backups to newer technology, but joe public is one day going to bring a CD out of his dad's attic and find he cannot even look at the photos on it.

    5. Re:Digital == Loss of freedom by aaronl · · Score: 2, Informative

      Actually... for the most part the formulas are publicly known for developers. You don't have to worry about acid fixes and rinse baths being proprietary. For most developers there are a number of companies that will make the chemical for you. Then you can just mix it yourself, or dilute it. I suppose you could make it yourself from scratch, but many of them use all sorts of evil to arrive at the final chemistry.

      Also, it isn't particuarly impossible to make your own film. Sure, getting the emulsion nice-nice isn't easy, but it's possible to do yourself.

      All that aside, you can develop most negatives from any vendor in the same chemistry. It's the C-41 process, and it makes industries like one-hour photofinishing possible. It also makes your life easier as an enthusiast.

      IOW, you don't have to worry about film being shut down because Kodak or whoever doesn't feel like manufacturing any more. It might be more difficult to get your favorite chems is all.

    6. Re:Digital == Loss of freedom by Phanatic1a · · Score: 4, Insightful

      There is always some risk in machine readable data.

      There is always some risk, period. If I keep stacks of negatives in boxes in my house, then in 20 or 30 years time my house might catch fire and burn to the ground.

      Okay, *I* might be clued up enough to always keep my negatives in a fireproof safe, but Joe Public is one day going to use a penny as a fuse replacement and find that he can't look at the photos on all that celluloid ash.

    7. Re:Digital == Loss of freedom by eggz128 · · Score: 2, Insightful

      On the other hand, in the analogue world, in 20-30 years Joe Public pulls his prints and films down from the attic and finds they've been damaged due to damp etc.

      There's always some risk no matter what you do. Call it the 'shit happens' principle.

    8. Re:Digital == Loss of freedom by Eivind · · Score: 2, Insightful
      That is a risk you can't fix anyway, I mean even if you where using a film-based camera the authorities could still take your pictures and negatives away by force.

      If anything digital makes it easier to guard against that, because it makes it trivial to ensure mutliple, backups. If you're *really* paranoid you make you've got atleast 5 backups in 5 different jurisdictions.

    9. Re:Digital == Loss of freedom by theAtomicFireball · · Score: 2, Insightful

      What bullshit. The lengths that some people go to to "defend" film. If you like film, use it, but don't come up with these lame arguments, just say you like it and be done with it. I've NEVER lost a digital file since I went digital back in 1998. I keep my photo library on a computer and do periodic backups to DVD, which I store offsite. In my years of using film, I did lose negatives (only one copy of them, remember?) had many negatives that could only produce poor quality images because they got scratched, faded, creased, etc., or required a lot of work to get a good print. At many times, I didn't have room to set up my darkroom (and most people don't have one), so I had the negatives, but no way to make copies except to pay someone else - making me dependent on someone else, which is far worse IMHO than being dependent on a file format you don't own. I only periodically shoot in RAW, but I have an "official" converter that will turn it into any number of open formats, plus Photoshop supports the RAW format. If in some weird stroke of luck, both companies revoked my licenses to use that software, I'd convert them all to an open format. Otherwise, my images are JPEGs or TIFFs. I can do anything I want with them and there's absolutely no doubt that I own them. I can make unlimited copies with no image loss without a darkroom, make backup copies to store elsewhere, and print copies from pictures taken years ago at exactly the quality they were when I took them. How in the name of Dog is film "better" in terms of image ownership? And how did this lame argument get a +5 insightful?

  3. What's been said before by twoshortplanks · · Score: 3, Informative
    --
    -- Sorry, I can't think of anything funny to say here.
  4. RAW format by James_Duncan8181 · · Score: 4, Insightful
    The RAW format wars seem an odd competiton between camera manufacturers, who are actively hurting their presence in the professional space by making their imafes less useful for archive purposes and less interoperable for press agencies to sell. The thing that is particually noticible is that manufacturers are now being actively co-opted into sharing this information under NDA with MS to allow the hardware to work seamlessly with Longhorn. This mass move from open to propriatory standards (something MS will, of course, encourage) is meaning that the camera manufacturers are seeing their poduct become commoditised, and apparently feeling unable to compete on hardware quality alone.

    Thanks Canon, you just made me finally feel confident about buying Taiwanese.

    --
    "To any truly impartial person, it would be obvious that I am right."
    1. Re:RAW format by Uruk · · Score: 2, Insightful

      I think though that most people buy cameras for the functionality of the camera, rather than the image file format. Besides, a lot of the cameras that have strange formats come with ready-made software to convert that format into something reasonable, (even if such software doesn't run on minority OS's)

      They aren't hurting their presence in the professional space - those folks still are going to buy the camera for its camera features. That provides an opportunity to sneak in other stuff that the camera company thinks will distinguish it from other companies, including different file formats with extensions to do particular things.

      is meaning that the camera manufacturers are seeing their poduct become commoditised, and apparently feeling unable to compete on hardware quality alone

      If what you say there is true, you've identified the reason for the strange formats and differences between cameras; they are trying to differentiate themselves.

      --
      -- Truth goes out the door when rumor comes innuendo. -- Groucho Marx
    2. Re:RAW format by DigicamGuy · · Score: 3, Insightful

      The thing that is particually noticible is that manufacturers are now being actively co-opted into sharing this information under NDA with MS to allow the hardware to work seamlessly with Longhorn.

      Actually, that's incorrect. The Longhorn interface is binary-only (no source code or format information is communicated to Microsoft or to the OS). Basically, the manufacturer (or third-party developer) writes a driver with an API that makes processed RGB data available to the OS. This is the same basic mode of operation as Canon and Nikon (and probably others) have implemented already in their free SDKs. Here's a brief interview with a Microsoft exec about the Longhorn interface and the shortly forthcoming "powertoy" RAW thumbnailer/viewer that's coming for XP. -- Not likely the level of detail /. people would want, but more than I've seen elsewhere, may help dispel some of the misconceptions.

      Of course, this means that the proprietary RAW formats remain entirely proprietary in the Longhorn era.

      For the record, I personally think that some level of open documentation of RAW formats makes a whole lot more sense than trying to come up with a common standard. A number of people (Adobe prominent among them, of course) have proposed Adobe DNG as a "universal" format. This sounds like a wonderful idea until you look at the assumptions underlying the format: It assumes a rectilinear pixel array, with a Bayer color filter array pattern (a checkerboard of RGB color filters on the pixels, with twice as many green pixels as red or blue). This is indeed the format used by the majority of cameras out there, but it completely misses innovations such as Foveon's full-RGB-in-every-pixel sensor, Fuji's hexagonal-pixel/diagonal-array "SuperCCD", and Fuji's latest "SR" sensors, which combine low- and high-sensitivity sensors in each pixel.

      While a "universal" RAW format would help with the issue of access to the underlying data, so would simple documentation of the structure of various proprietary RAW formats, and the latter wouldn't have the negative effect of stifling innovation in sensor technology.

  5. Re:Can anyone say... by meringuoid · · Score: 4, Funny

    Somewhere in the world where the Americans aren't in control? Closest you're going to get is probably Iraq...

    --
    Real Daleks don't climb stairs - they level the building.
  6. How Open is the Repository? by Anonymous Coward · · Score: 4, Interesting

    This effort is being set up by a guy (Juergen Specht) who hosted a mailing list and then deleted it without notice when some of the posts offended him.

    See:
    http://www.vudeja.com/04/09/mailing-list

    http://www.esthet.org/blog/archives/001294.html

    http://www.wirefarm.com/archives/004186.html

    http://www.easterwood.org/hmmn/archives/001111.htm l

    http://openraw.org/about/

    Don't be surprised if this site just up and disappears one day, taking all of the data with it.

  7. Re:why? by BabyDave · · Score: 2, Informative

    I think it refers to the camera's internal representation of the image (i.e. the "raw" data)

  8. What horseshit by loraksus · · Score: 3, Insightful

    The manufacturers are just opposed to working together to create some sort of standard.

    But can you blame them? Really, think about this for a second - people (scumbag fucks who should hang from lampposts, call them what you will) from Rambus sat in standards groups for years and then turned around and secretly patented the standard and then had the balls to demand royalties. You saw more or less the same bullshit with .gif and in hundreds of other similar cases in the last 20 or so years.
    I think it is (sort of) understandable that companies would be hesistant to work together to develop a standard way of doing something - especially in a cutthroat business such as photography.

    And by the way, using Canon is a fairly shitty example, Nikon is far worse when it comes to the RAW format (ok, its not really a format) bullshit that flows through the world of pro photography.

    That all said, this smacks more of the petty bickering that is involved in cameras more than than anything else (See Also, "Complete lack of lens interchangability" et al), but as always, we (or those who buy $600+ cameras) get fucked.

    Don't get me started on how "using the DMCA to "protect" the super complex almost but not quite encrypted raw format". I don't need a stroke at this age. . .

    --
    1q2w3e4r5t6y7u8i9o0pqawsedrftgthyjukilo;p'azsxdcfv gbhnjmk,l.;/
    1. Re:What horseshit by mukund · · Score: 2, Interesting

      The manufacturers are just opposed to working together to create some sort of standard.

      Adobe made an open format called digital negative... The camera manufacturers need to start adopting it.

      --
      Banu
    2. Re:What horseshit by Hast · · Score: 2, Informative
      Unfortunately DNG doesn't go all the way and just moves the problem further down the line. From the OpenRAW FAQ:

      DNG also allows "private data" to be stored in the DNG file. This private data is only known to the camera company that wrote the private data. Third party software that reads and/or writes DNG files will ignore private data recorded by the camera. Only the software written by the camera maker will read the private data written to the DNG file by its camera. Some of this private data might be important or useful information needed by a RAW converter. Adobe's DNG format does not eliminate the problem of undocumented RAW files but transfers the problem into another "container", the DNG file. By allowing private (undocumented) data in the DNG file, DNG does not meet OpenRAW's goals.


      So in the end you end up with a similar situation. You can read the data but you don't know exactly how to treat it.
  9. So does basically every camera by Jonathan · · Score: 4, Informative

    The point isn't that you can get a jpg out of your camera -- I haven't seen a digital camera that can't -- the problem is that the original, uncompressed data generally isn't in an open format.

  10. Please tell me this, this is critical.. by johansalk · · Score: 2, Interesting


    From Canon, as they refused to cooperate with openRAW and ended their letter with a slap in the face: "If our equipment or software does not meet your needs, you are entirely welcome to seek other suppliers".

    And this is *exactly* what I'll do from now on and for the foreseeble future; I will *not* entrust the future accessiblity of my visual data to such a company and its formats, and I will not render myself under their mercy given their manifest chauvinism. Does anyone know what suppliers are cooperating with openRAW? Those will get *all* my business.

    Thanks

    1. Re:Please tell me this, this is critical.. by Ubergrendle · · Score: 3, Interesting

      From Canon, as they refused to cooperate with openRAW and ended their letter with a slap in the face: "If our equipment or software does not meet your needs, you are entirely welcome to seek other suppliers".

      This is true, and unfortunately Canon can afford to take this position. In the DSLR market -- the *serious* digital photography market -- Canon has through various reports a 50-70% market share. Their only serious competitor is Nikon who controls anywhere from 30-50% depending upon who you listen to, and the rest make up a very small percentage. Kodak just announced a complete retirement from the DSLR market, Sigma cameras are doing horribly, and although Pentax and Minolta have decent offerings their market penetration is relatively weak. Canon can throw around threats since Nikon is WORSE in their disregard for RAW, actively encrypting (weakly) the white balance data. Nikon knows the encryption is a joke, but its enough to have legal teeth via the DCMA and thus Adobe won't translate it.

      Personally I'm more concerned with the retirement of RAW formats than the current vendor specificity. When you by a Canon EOS system or a Nikon F-mount you're buying into a closed, proprietary hardware system. Extending it to the software realm is crappy, but not surprising. Microsoft is best positioned to bust this wide open, and its in Adobe's best interests to open RAW or see the success of DNG. My guess is once the balance of power starts shifting heavily in favour of Canon or Nikon (towards virtual monopoly) the lesser company will open up their RAW format to be more accomodating.

      --
      John Maynard Keynes: "When the facts change, I change my mind. What do you do?"
  11. Need for a broader approach? by archeopterix · · Score: 5, Insightful
    Hm... Everytime I read something similar to the article (that is about twice a week), I think that there should be a non-profit organisation to oppose taking away customer rights under the guise of "intellectual property".

    Something powerful enough to organise boycott that would cause *pain* to the offending company. Something that a congresscritter would be afraid to piss off. EFF comes close, except that it a) has a broader scope and b) sadly is not powerful enough.

    Too bad that the existing consumer organisations are focused on making money from their "consumer reports" and the general population doesn't care (the frog is half-boiled and still comfortable).

  12. going canon fanboy in here but by william_w_bush · · Score: 2, Informative

    so far my experience with canon gear has only been positive. firstly, they released linux and osx drivers for an old inkjet i had, along with most of their current line, and a good number of their scanners. second, their cameras are amazing, and use normal sd and cf cards instead of the MS and XD that are becoming infuriatingly ubiquitous. Also, their printer lines tend to standardize on the same types of ink, with better quality than the hp's and terrible machines epson is putting out nowadays (my r300 photo printer ran low on lt yellow ink, so it won't print black and white and keeps nagging me in windows to order more ink).

    i suspect this is just canon usa marketing dicks playing bs politics for their own sake. so far theyve given out a lot better specs for most of their printers than most companies, and few printer mfg's will even bother to put out cups drivers for their lines.

    not releasing their RAW format seems amazingly petty, but sounds exactly like all those fat, middle-aged sales execs who thought it wasn't worth it developing open-sourced linux drivers, cause they could get more commision charging each customer for the drivers themselves. we released them anyway, but a lot of those types make VP and do stupid shit like this to try to throw their cock around.

    --
    The first rule of USENET is you do not talk about USENET.
  13. Re:why? by croddy · · Score: 2, Funny

    oh, they do. just add -me to your search terms :-)

  14. its not so much digital thats the issue... by petermgreen · · Score: 2, Insightful

    they could just use an unecumbered format like png

    BUT the advantage or "raw" is its the closest you can get to what actually came out of the cameras CCD. because of the way CCDs work this will be about a third the size of the resulting image (assuming they are uncompressed or compressed using a lossless algorithm that gets roughtly the same compression on both).

    --
    note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
  15. Re:why? by petermgreen · · Score: 4, Informative

    in this case (cameras) raw reffers to the fact its the raw unprocessed data from the CCD

    this has to be processed to convert it to a form that we would recognise as an image file. This can happen either on the camera or on a PC.

    However This conversion process may well not be fully reversable (due to rounding errors) and bloats the data considerablly (CCDs generally make a red green OR blue value at each location image files generally have red green AND blue at each location so turning CCD output into an image file always involves interpolation) so from an archivists point of view its best to keep the raw data unfortunately that raw data is often in a closed format.

    --
    note: i'm known as plugwash most places but i screwd up registering that here somehow in the past and now can't register
  16. Just keep... by troon · · Score: 2, Informative

    ...a copy of the dcraw source code.

    --
    Ydco co ,df C erb-y go. a Ekrpat t.fxrapev
    1. Re:Just keep... by RDW · · Score: 2, Informative

      The various manufacturer-specific Perl modules in the ExifTool package are also an excellent source of documentation for RAW file metadata. Reading this (rather well-commented) code can help make the more cryptic dcraw source much more comprehensible.

  17. Re:why? by croddy · · Score: 2, Interesting

    well, yes -- that page calls it "Raw" and "raw", which makes sense. I'm just confused about the "RAW" nomenclature.

  18. Re:storing raw digital images is stupid by jgordon7 · · Score: 3, Insightful

    Most raw formats on modern cameras is compressed using a lossless format. Take Nikons nef format, a 6.1mp image in raw is only about 4-5 MB, if this was not compressed it would be closer to 10 MB.

    You can not tweak certain settings as easily once the image is convert to another format, even a lossless one like tiff. Best example is white balance.

  19. you need to know what you are doing by cahiha · · Score: 2, Informative

    By converting, for example, to JPEG and throwing the RAW away you are losing lots of creative post-processing control.

    Indeed, you are. That's because JPEG doesn't have the depth and gamut to represent digital images captured by modern cameras. If you use a more modern cooked, open format (e.g., TIFF, JPEG2000) at the right depth and at a lossless or high quality setting, then you lose nothing.

    Later you may want to re-process the picture with superior software (it does get better over time).

    And you can do that because the interpolation from raw to RGB is invertible given a tiny bit of metadata (metadata you would need to carry around with the raw image as well). If you are dealing with a four channel sensor, you can still store the data in a four channel TIFF. No need to store RAW.

    Or perhaps you just needed to tweak the white balance.

    You can do that in RGB.

    Or fiddle with the sharpening.

    For professional archival applications, you should turn off in-camera sharpening. That has nothing to do with whether you use raw storage.

    Or the tone curve.

    You can do that on the cooked image just as easily as on the raw.

    I think the whole confusion arises because people like you equate non-raw storage with JPEG. JPEG is a bad choice for high-quality archival storage, but formats like TIFF and JPEG2000 are fine.

  20. Summary of some key points: by Distan · · Score: 3, Informative

    What is this "RAW" format?

    RAW doesn't really refer to any single file format. RAW refers to pulling the unprocessed (raw) sensor data out of a digital camera. The actual layout of the bits varies from brand to brand, and often from model to model.

    Why do photographers want access to the raw data anyway?

    Many professional/prosumer photographers like to archive the version of their work that contains as much of the originally captured information as possible. In the professional film world, this meant processed slides (for consumers, this meant processed negatives). In the digital world, the RAW file contains all the data captured by the camera, before some data is lost by compression and other data is added through interpolation.

    Can't they just pull a lossless image out of the camera and be happy?

    No. The very act of converting the raw data into an image involves lossful processing of the data. Out of gamut color data is discarded, and CCD color data is interpolated to fill surrounding pixels.

  21. There is no real alternative.. by shaanX · · Score: 2, Insightful

    when it comes to doing any sort of real editing JPG is *NOT* an option. I see people here that are obviously not photographers or have not used a digital slr camera saying just to use JPG. If you plan on doing little to no post processing sure use JPG but for true manipulation RAW is essential