Slashdot Mirror

On the plus side, raw support is pretty good with free alternatives

That's because everyone (including Adobe) uses the exact same RAW library. It happens to be open-source, too.

DCRaw is a raw image handler for basically every camera out there. It's such an impressive piece of code everyone uses it,

Gimp won't. Use dcraw...

This utility converts the native (RAW), format of various digital cameras into netpbm portable pixmap (.ppm) image. Supports the following models: Canon, Kodak, Olympus, Nikon, Fuji, Minolta and Sigma (see http://www.cybercom.net/~dcoffin/dcraw/ for full list) Note: This utility does not read directly from the cameras, only the files after they have been downloaded, use gphoto2 for that.

I wonder if perhaps the noise reduction and stealth features came at a price of reduced performance.

Any helicopter can enter a state known as settling with power or vortex ring state. Basically the rotors downwash is recirculated around and back through the rotor. When operating close to high walls, a vortex ring can form very suddenly, and the pilot may not have sufficient room to recover.

Here's a link describing the condition.
http://www.cybercom.net/~copters/aero/settling.html

Not necessarily a limitation of any modifications, just a general limitation of helicopters.

I knew that helicopter ground school would come in useful eventually! :-)

Realistically here, the high end camera makers don't write plugins for the GIMP, so if one wants to make use of the RAW images from one's EOS-1 or other camera without losing data, they are either using Photoshop, or perhaps Lightroom.

Nikon and Canon don't write plugins for Adobe products, either. Adobe writes them, and they don't even have full access to RAW specs, as camera manufactures keep them proprietary and secret. Most of it is reverse-engineered, with some (unknown) data simple being unused by Adobe products.

If GIMP developers went through the same effort of reverse-engineering formats, they'd be able to support them, too. Although, I still don't see many serious professionals using GIMP -- the difference in other features and performance is just too great.

There's no need to reverse-engineer anything. Anyone involved with processing photos already knows about dcraw which is used to turn RAW files into something usable. In fact, Adobe uses it in Photoshop, as does practically everyone else except the camera manufacturers.

Now, they all modify the code as they think they have a better Bayer interpolation code than everyone else, but that's their perogative. (RAW images are just raw sensor data plus metadata).

http://www.cybercom.net/~dcoffin/dcraw/

GPLv2

There is a gimp plugin that provides a very nice front end for it.

They've been in Debian for years.

There's an open source project (Dcraw which aims to solve this problem.

The source code file can be found at this file

The digital camera market has standardized to RAW

They have? Which one?

The situation is not so good for professional-grade equipment which very often use proprietary format to store hi quality pictures (each different series from each different manufacturer use their own home-made format for "RAW" pictures). Very often those format are poorly documented, kept secret or protected from reverse-engineering by DCMA. They are near to no tool to handle them (appart from the software that came with the device). In 30 years, the knoledge about one peculiar format may very well be lost, and no more software could be found that can open it (and pretty much sure that, had that software be excavated from somewhere, the deprecated OS and hardware running it will be missing too).

i started converting all my RAWs to DNG. it's already being adopted by the biggies and even though adobe hasn't made their converter open source, dcraw has.

i'm now making plans on how to convert the 15,000 RAW files that i have on various DVDs and external drives to DNG without losing all the meta information in Lightroom, but that's another topic...

Absolutely.

Since Dave Coffin's dcraw utility is open-source (in fact, I think it's public domain) there is no reason why it would vanish in such a short time. You will be able to find a compiler somewhere since it's written in C.

Now, if you were relying on the propriety closed-source software that came with the cameras you'd be in trouble, but Dave's software is generally better quality than that half-arsed crap anyway.

'RAW' isn't used by anybody. 'RAW' does not exist. 'RAW' is a collective name for a shitload of formats by a smaller shitload of digital camera companies.

No, it's not. RAW = Canon's "raw" image format. "Raw" image formats are produced by many higher-end digital cameras. I'm sorry you don't understand the distinction between RAW and raw, but it does make it painfully obvious this isn't your area of expertise. It is mine: I've shot RAW images on my Canon dSLR for fun and profit for several years now. I shoot exclusively in RAW format because of the extra bit depth which makes adjustments much more 'transparent' (a level adjustment won't cause as much problems wit 10-12 bit data as it will with 8 bit, and you also have no compression artifacts.) I archive everything in the original Canon RAW format.

Since you claim expertise in this area and make some arguments that are, on the surface, convincing I feel it is important to point out mistakes in your arguments. Mistakes that even a relatively raw beginner such as myself are aware of. It appears that you have a very high level understanding of RAW, but to extend this into an understanding of the internals is a dangerous thing to do on Slashdot. First of all, since you speak of Canon, Nikon, Fuji, Pentax 'raw', I think you do understand that each of these formats are unique. The original poster is correct that some manufacturers (e.g. Sony) actually encrypt some of the data in their RAW format so that (for instance) the white balance can only be extracted using proprietary software. It may not be "a Sh!7l0ad of smaller manufacturers" from your point of view, but since I've seen relatively inexpensive Sony, Canon, Pentax and (the dearly departed) Minolta cameras spit out what their marketing material claims is "raw". The bottom line is that RAW is like tiff, only worse in that the data, data representation (byte order...), encoder and container may change from manufacturer to manufacturer. The only thing Canon RAW and Sony RAW are certain to have in common is that their marketing material, instruction book and camera's menu uses the three letters 'R', 'A', and 'W' to represent the name of the format (or in some cases 'r', 'a', 'w'. For a close look at the internals of many raw formats, I suggest you look at the source code to Dcraw. A few other mistakes, even really cheap webcams don't encode to gif (I don't know where that comment came from but they don't, the closed driver software takes the "raw" CMOS/CCD data and encodes it to GIF without letting the user see the raw data. If you're into astrophotography or have used a webcam on an opensource operating system, you'll understand more. Also, the raw file may be the closest consumers can come to the CCD's internal format, but by no means does is it identical to the RAW CCD data as it comes out of the CCD's analog light buckets (or CMOS gates) into the A/D.

Since I shoot RAW, I also burn a copy of dcraw.c onto every disc - so even if the current platforms get lost by the wayside, there will be code to convert them still.

Storage itself? Currently burning onto Delkin Archival Gold, storing cool and dark, and in two physically distant locations.

They're also stored on my harddisk, and the best are backed up onto a USB drive.

If it looks like the DVD-ROM drive is becoming obsolete I'll burn them on to whatever comes along next.

If you're truly paranoid you can always print them on archival quality paper using pigment based inks ;)

As I understand it, the Nikon DSLR bodies will accept old F-series lenses, but they will require manual focusing. Canon EF-series lenses for the EOS system will work on the Canon DSLR bodies. Older third-party EOS system lenses (e.g. Tamron) may need to be rechipped for use in Canon DSLRs.

Canon's .CRW format includes a large JPEG "thumbnail" in addition to the raw image dump. This can be useful as a rough guide during postprocessing.

Dave Coffin's dcraw program will decode a wide variety of raw image formats, including Canon, Nikon, and Pentax. It is also used within the ufraw package, which can operate in a standalone mode or as a GIMP plugin.

Don't say The Gimp -- it can't edit RAW images.

This is not strictly true, there are a few plugins available for RAW image support.

UFRaw
There is also a basic plugin available from the author of dcraw.

What options are there to edit RAW photo files under Linux?

As with all *nix stuff, the RAW handling is done by a separate component. Investigate UFRaw and DCRaw. UFRaw even has a plugin for the GIMP that works well. As an amateur photographer I use and highly recommend UFRaw.

Using dcraw and the Radiance (HDR) file format, it should be trivial to convert any digicam or SLR's raw image to an HDR.

For manually-captured bracketed images, there's AHDRIC (disclaimer: I wrote this). As long as the EXIF info is intact and the only thing that changes between shots is the shutterspeed, this should do the trick. A related tool (AHDRIA) lets you capture HDRs automatically by controlling a digicam via USB (Canon digicams only, sorry). This process can take 20-120 seconds, depending on the quality required.

'GIMP is probably never going to be able to support your RAW photos'

There's already an excellent solution to this problem:

http://www.cybercom.net/~dcoffin/dcraw/

It's been used to create several GIMP plugins, including:

http://ufraw.sourceforge.net/

When run in 'standalone' mode, UFRaw even has a 16-bit output mode (useful if you Cinepaint). When run as a GIMP plugin, you're obviously stuck with GIMP's 8-bit limit.

Did you try the rawphoto plugin? http://www.cybercom.net/~dcoffin/dcraw/. 30 seconds with google gives a lot of options.

The biggest problems with taking all your photos in this way are that they're large (very large, relative to JPEG), and you either must use proprietary camera software or plugins for software like Photoshop
For the first part of this statement, I find that when I take pictures with my 20D the difference in size isn't that big. It's like 2MB for JPEG and 6MB for RAW. (I had a couple of shots taken in JPEG(Large, Fine)+RAW). If you want to do more advanced stuff like noise surpression then RAW is the way to go.

For your second statement there is a OSS RAW converter out: DCRAW. AFAIK it produces better quality images than most software you get with cameras. (Though it may not be as good as the professional tools.)

You should know about this, there is also a nice interface. Many cameras should be able to do time lapse themselves.

The Nikon WiFi support with the D2X does support standards, FTP and PTP/IP, which are both published and supported by Linux.

The ptp/ip protocol:
https://www.fotonation.com/
Linux support for digital cameras and PTP/IP in particular:
http://www.gphoto.org/
Raw image processing, including encrypted Nikon D2X images:
http://www.cybercom.net/~dcoffin/dcraw/
http://www.robgalbraith.com/bins/content_page.asp? cid=7-6459-7213
http://www.photoreview.com.au/Articlexasp/90c83053 -0a7f-45cc-ba68-9560e9f3c061/Default.htm
http://www.mobilemag.com/content/100/336/C3218/
http://dailywireless.org/modules.php?name=News&fil e=article&sid=3061
http://www.eetimes.com/news/latest/showArticle.jht ml;jsessionid=E0TTJLUSVT5NSQSNDBGCKHSCJUMEKJVN?art icleID=47204433&_requestid=171509

...a copy of the dcraw source code.

Already done. Check out Dave Coffin's dcraw code available here:

dcraw

But he's in Mass. But there are copies of the code outside of the US already, so Pandora is free and won't go back in her box no matter what.

I wonder if we should start referring to "coffin" in Nikon's nail now?

Nikon needs to get a clue and give us, their customers (I have a lot of $'s tied up in Nikon DSLR gear, and Photoshop/etc. too) what we want. Documented RAW formats and/or support of an open raw standard like DNG.

Acording to the author of dcraw it's not only Nikon who are encrypting their metadata...

I just looked at dcraw.c and the parts pertaining to parsing Canon's white balance info simply use the camera model name to determine where in the RAW file Canon put the WB. Hardly "encryption", it's just an offset that varies by format.

Canon appears to develop a unique RAW file format by camera model. That makes a "tiny" bit of sense in that each file can accurately describe precisely the data the camera is capable of producing. It makes it harder in the long run to support dozens of file formats, but that's a trade-off Canon appears to be willing to live with. Keep in mind that Canon has to eat their own dogfood, too -- every format they produce means a new software release to parse the RAW files. And Canon doesn't charge for these downloads -- once you've bought their camera, it comes with software and upgrades (so far) have been free. So there's no real economic incentive for them to continue this, but they do.

What I think is most important regarding this issue is that it's simply a tempest in a teapot, being stirred by Adobe for their own political reasons. First, it's only on a single high-end pro camera -- affecting only a select set of professional photographers, most of whom have never heard of Open Source. Second, it's only white balance information. It's what the photographer told the camera about "white" or "gray" at the time of the shot, but it doesn't change the underlying image data. It's nothing that can't be recovered in the digital darkroom during processing. Finally, the encryption is trivial to break -- Adobe is raising a ruckus claiming the DMCA is preventing reverse engineering. In reality, most Open Source developers would simply ignore the DMCA and perform the decoding anyway.

In the camera world Nikon stands alone in this stupidity, but it's really too small of a matter to concern any of us, (unless you're looking for a DMCA poster child to nail to the wall.)

application for the Nikon SDK is possible for bona fide software companies that send Nikon a written application for the SDK. Once approved, the SDK is provided to the developer at no charge and they are authorized to use it.

One key difference is that Nikon has not only left their file format undocumented, they've actively encrypted a key image parameter, allegedly as a spoiler tactic to prevent 3rd party developers fully parsing the files without signing up as 'approved' developers. If Nikon decides you are a 'bona fide' software company worthy of the honour, you can get hold of an SDK (apparently Windows/Mac C++ only with binary runtime libraries) but won't be given a full description of the file format. This has serious implications for the use of Nikon NEF files as an archival format (will Nikon's SDK components work on whatever OS you are running in 20 years time?), for developers who want to use their own algorithms (like Adobe), and for FOSS projects. Luckily, Dave Coffin has already reverse engineered the decryption algorithm in the current version of his open source dcraw RAW converter, so we're not yet locked out of the NEF format. What isn't yet clear is whether Nikon will continue with this sort of tactic in future NEF versions, and if Adobe will overcome their DMCA concerns to fully support NEF in their ACR raw converter (assuming they're not just grandstanding). Incidentally, there's a brief description by Tom Christiansen of the white balance encryption algorithm here, and a pointer by Thomas Knoll (of Photoshop fame) to the relevant section of the dcraw code here.

Looks like Nikon's goofy encryption has been broken.

Oh, here's a link to dcraw which will blast through Nikon's bullshit.

Fortunately, outside the Land Of The Free(tm), anyone can access Nikon's encrypted data with a simple GNU/Linux application

dcoffin:Format no more secret

Dave Coffin's open source dcraw RAW converter has been mentioned here a couple of times, but it's worth pointing out that the latest version already supports decryption of Nikon D2X white balance data. There's a brief discussion of the decryption algorithm here, and Photoshop developer Thomas Knoll points to the relevant section of code here.

As far as I know, Nikon has not threatened to invoke the DMCA over decryption of the data - this looks more like a rather lame attempt to obfuscate one of the key image parameters to make life difficult for 3rd party developers (who Nikon can then claim have 'incomplete' support for the NEF format if the camera's white balance settings can't be extracted). Of course this doesn't rule out the possibility that Nikon is deliberately playing games with Adobe, since the Photoshop developers have to choose between implementing decryption that might be actionable under the DMCA, and leaving out support for 'as shot' white balance (it seems like they've gone for the latter, for now). Smaller fish like Bibble are already including white balance decryption, which could leave Adobe at a disadvantage if they continue to believe that this is a legal risk (and who knows the details of the DMCA better than Adobe?!). Meanwhile, Nikon presumably hopes to sell more copies of Nikon Capture (though to be fair, Nikon View, which even comes with a rudimentary Photoshop plugin, can also handle these files and is a free download for Nikon users).

Ahem. Apparently the D70 actually is readable by dcraw so I may be able to use it (and possibly the D70s as well). I guess Nikon only cares about restricting its tip-top of the line cameras, which I can't afford. This doesn't change the fact that I don't like how they are acting, though.

I don't know what software Adobe has written regarding this, but the way I figure it, that is irrelevant. The DMCA is not what stops Adobe from providing their users with the kind of access to Nikon raw images that their Nikon-using Photoshop users may want, although I experience a good schadenfreude laugh at Adobe's expense when I read people make the argument that Adobe is somehow disadvantaged by the DMCA here.

Adobe can use some of the money from distributing proprietary software (ill-gotten gain, in my opinion) to negotiate a binary-only proprietary copy of a Nikon library to link to Photoshop which would allow Photoshop users to decrypt the Nikon raw white balance segment that is encrypted.

Nikon and Adobe both walk away getting what they want: Nikon's encryption is no less "secure" than it was before (how this works can be hidden from hidden from everyone, including Adobe), and Adobe gets to supply plug-and-play functionality to their users. Meanwhile, and more importantly, their users are left without their software freedom, and no ability to easily deal with Nikon raw images in other programs. Those users are paying their money in exchange for a loss of their software freedom and complete control over their images. If Adobe complains about not wanting to ship software under its name without full and complete source code to that program so they can inspect, modify, and even share it should the need arise, I'll get another schadenfreude chuckle at Adobe's expense because I want software freedom too. The only difference is I don't want to hoarde it from others.

Both Adobe and Nikon are treating their customers like dirt and it's always a good time to remind Adobe that it was wrong to have Sklyarov arrested, detained, and subject to an "agreement"..

To me, these are all excellent reasons to avoid or stop doing business with both Adobe and Nikon. Run the GIMP, use some other brand of camera that will give you the unencumbered raw functionality you want, and retain full control over your photos. The dcraw website hosts a list of cameras it will work with. Surely some camera on that list will meet one's needs without going to Nikon.

http://www.cybercom.net/~dcoffin/dcraw/

dcraw supports more raw formats with a handy command line interface, and runs on Linux and Windows to boot.

And the good burgers from Nikon intend to prevent direct access to crucial parts of the raw data of my images?

Stop jerking that knee. There was no "encryption" worthy of the name. It's been "broken" already, and the necessary function is already available in open source. If you read between the lines in the posted article, it sounds like Adobe hasn't even asked Nikon for permission -- possibly because Adobe thinks it has more to gain by fucking with Nikon in the court of public opinion and wants to be quite sure it doesn't have to offer any in-kind promises of immunity from its own IP abuse.

If Nikon had sued anyone over this, or even claimed the right to sue, they'd deserve condemnation. They've done neither. The original article is in substance an Adobe press release, and ought to be regarded as such.

As Adobe themselves state, they expect to have preliminary support for the D2X in May ( http://www.adobe.com/products/photoshop/cameraraw. html )

As the article itself mentions, it's not really that big a deal. It is the white balance parameters as set on the camera when the image was shot that is encrypted. The RAW data isn't directly affected by this, and picking a white-balance preset or performing manual/auto whitebalacing on the RAW data gives you the same/similar/better results (that's partly the point of shooting RAW, no?)

What could be worse is if they encrypted the data as well. This is what SONY does on the F828 and V3, for example.
However, both are supported by Photoshop's RAW support, so I take it they simply licensed or SONY gave them a thumbs up for supporting it. No idea why they encrypt it, though.
Regardless.. that's what would have to happen with any future encrypted formats.. I doubt we've seen the last of them anyway.

If all else fails, get the dcraw utils ( http://www.cybercom.net/~dcoffin/dcraw/ ) and convert from one raw to another. That's where the SONY decryptor is also hosted.

Speaking of dcraw... has Adobe given the author credit yet ?

"There are no other products. Gimp (which I prefer in many cases) can't do half of the things a professional graphic artist needs"

Depends on the artist between Gimp and Cinepaint (a gimp fork) many video editing houses use Gimp for single-frame edits because it's cheaper, lighter weight, and more easily customizable than PS.

"And when looking at photo editing, I havent' seen ANY product that has good RAW support"

I do all my RAW file editing using Gimp 2.2. There are plug-ins that you need, and a stand-alone RAW tool, but the documentation for installing is quite easy to follow, even for less advanced users.

There are also some camera-specific RAW tools that go a step beyond, taking advantage of many features which are extensions to the basic RAW file support.

The Foveon X3 Capture chip requires a different kind of interpolation. Unlike CCD arrays, it captures three colors at every pixel location. But the colors are not well separated, so the raw data looks very gray. Enhancing the color without also enhancing noise is very difficult.

To this day you can get all sorts of open source digital camera software that supports all sorts of competing cameras from that era, but NOT the DC-50.

Then you haven't done your research very well (or mine is at fault). dcraw claims to support your camera (and practically everything else, including the 300d/DRebel/etc mentioned on this thread).

dcraw is a great little RAW decoder http://www.cybercom.net/~dcoffin/dcraw/

And if you go down to the "Commercial Software" section, you'll see that it or parts of it are used by quite a few people.

Here you go...
Raw Digital Photo Decoding in Linux

Well, raw files aren't in RGB - they are a dump of the sensor readings. I can't find a reference anywhere, but it looks like each site on the sensor is just sensitve to intensity, not any particular color.

A grid of filters is placed over the sensor so that each sensor location receives a different color. The camera (or software if working with raw files) interpolates the other color values from it's neighbors.

I got this info from the "CRW Files" section of this page. It's written by the author of dcraw, which converts a variety of raw image formats into RGB.

Well, raw files aren't in RGB - they are a dump of the sensor readings. I can't find a reference anywhere, but it looks like each site on the sensor is just sensitve to intensity, not any particular color.

A grid of filters is placed over the sensor so that each sensor location receives a different color. The camera (or software if working with raw files) interpolates the other color values from it's neighbors.

I got this info from the "CRW Files" section of this page. It's written by the author of dcraw, which converts a variety of raw image formats into RGB.

I am using Nikon D70 and to convert RAW photos I have to buy some $400 Nikon application (or transfer photos manually with their free app).

Raw Digital Photo Decoding in Linux
RawPhoto GIMP-2.0 plug-in

The first is a command-line utility for processing the raw format images from a variety of cameras, including Nikon's NEF format and the second is a great plug-in for Gimp that integrates it with a GUI.

While I don't think your evaluation of the effects of the standard is necessarily wrong, I do think your estimate of the amount of "reverse-engineering" Adobe needed to do to support different raw formats may be off. That's because, as far as I know, much of the support for these formats is actually handled by software that Adobe didn't even write. Instead, it was written by Dave Coffin (http://www.cybercom.net/~dcoffin/dcraw/index_en.h tml) so he could edit the images from his Canon digital camera in Linux.

Adobe can save money with the adoption of DNG - but more money will be saved by smaller developers who cannot do the ongoing reverse engineering that Adobe does to support new RAW formats.

It's unclear to me how much effort Adobe is actually spending on the problem. My understanding is that just about everyone uses Dave Coffin's dcraw program to one extent or another. (Mr. Coffin reports Adobe as a company that uses at least part of his program. I don't know if they use his interpolation, but I'd guess that they use his decoding work.) Of course that program is available under a very liberal license, so all the little guys can make use of it, too. A bigger issue, IMO, is that if a camera uses DNG then it will be supported as soon as it's released, rather than requiring a software update before people can read its raw files.

Linux users can use the dcraw util to convert RAW into TIFF format. It also has a plugin for GIMP which works fine. On my camera though, the RAW files are 6.3Mb, and the TIFFs created with dcraw are 18Mb.

Have a look at my pics, too. :)

Most camera manufacturers provide converter software as part of the package, and in some cases it's of a higher quality than the stuff built into the camera. This shouldn't be surprising, since there are fewer time and processing power constraints in off-line processing.

AFAIK, though, the best available software is Dave Coffin's dcraw program. It's available as free software under a non-advertizing BSD-style licence. It can be used either as a standalone converter (with Windows, Mac, and Linux versions available) or as a GIMP plugin. The author also claims that the program, or at least parts of it, is used in many commercial programs including Photoshop. I've been pretty pleased with the results so far.

Try Emacs !Once you get your key bindings right, it's really a fabulous editor. Lisp grows on you FAST.
Since you're an Emacs person, Brad Merrill, a Microsoft .NET Evangelist, has some tools for doing C# in Emacs. Also the people at @ic#code have a free as in beer, but still open-source IDE for C# programming. I don't know if I can do straight C/C++ with it. The screenshot has me believing otherwise. It looks like Visual Basic. I'm thinking I'll give that one a try before tackling Emacs.

While we're on the subject, I've just found that Macromedia's Dreamweaver MX 2004 works as an IDE for C# programming. It does color syntax highlighting, which is an important feature for me, but I haven't found any other standard programming IDE features.

Finally, not being an Emacs guru, I'm not sure which implementation to get. I can always use Cygwin unless there's a regular Win32 port. However, I also found a Windows version that claims to be smaller and lighter at http://www.notgnu.org/ Any recommendations?

http://www.cybercom.net/~seishino/specs_header0806 2004.jpg

Take a look at Dave Coffin's CRW.c - a RAW reader that provides support for many formats. In doing so it ALSO tells you where the pixels are, such that you could get at them for modification... I'm working on a program for the SD9 RAW images that I am hoping to let users make some basic modifications to image data (like cleaning up some noise only in one channel).

The trick with most RAW formats is that since they are mosaics of color you would have to re-mosiac whatever you want to change, so my estimate of "pretty damn easy" is perhaps a little too strong - but once you know where the pixels are in the data and the format they are stored in (like compressed with table lookups) then it's game over for anyone even slightly determined.