It’s common to talk about “raw files” as if they were a single standard like JPEG. But nearly every camera capable of storing the unprocessed data from its sensor on a memory card actually uses its manufacturer’s unique proprietary format, which may even be specific to that model. This may not seem much of a problem, as those cameras almost always include a CD with raw conversion software packed in the box. There are also third-party raw converters that are often better than what the camera manufacturers provide.
The babel of formats could cause real trouble sooner than you might think. Like other forms of computer technology, digital cameras are evanescent. In a mere five years, Canon’s D30— a revolutionary 3-megapixel DSLR introduced in 2001— was successively replaced with the D60, the 10D, the 20D, and the 30D (as of August 2011, the current version is the 60D). That’s just one manufacturer’s lineage, and doesn’t even include Canon’s professional DSLRs or the Digital Rebel/300D and its successors. In 2005 Canon dropped support for the “legacy” D30 in their Digital Photo Professional raw converter. Complaints from the many people who still used the D30 forced Canon to restore that support. But who’s to say they won’t drop it again— perhaps along with the D60 and 10D— in a future version?
Even if you’re wealthy enough to regularly replace your cameras with the newest models, the ghosts of cameras past will still linger in your library of raw files. You could save the raw conversion software that came with those cameras and reinstall it when necessary. But what if it’s not compatible with your new computer or operating system?
The 64-bit versions of Microsoft’s Vista and Windows 7 operating systems can no longer run 16-bit legacy software written for DOS or Windows 3.1. And some 32-bit software that runs on Windows 95, 98, or XP is incompatible with the security enhancements in Vista and Windows 7. Apple’s Macintosh computers have used three distinct, mutually incompatible types of processors. The Lion version of the OS X operating system released in 2011 no longer includes or supports the “Rosetta” emulator that let current Intel-based Macs run legacy software for the earlier generation of Macs that used the Motorola/IBM PowerPC processor. So it’s not merely conceivable but very likely that 20 years from now, computers capable of running your old 32-bit Windows raw converters will be found only in museums.
Adobe is attempting to solve this problem with DNG (pronounced ding, but sometimes dingy or dinghy), the Digital NeGative format. DNG is a publicly-documented specification of a standard format for storing raw camera data. Adobe is actively “encouraging” camera manufacturers to abandon their proprietary formats and adopt DNG. Until that happens, they’re promoting DNG as an archival format for photographers. They’ve had a notable success with the American Society of Media Photographers’ Digital Photography Best Practices and Workflow project, funded by the Library of Congress. The project has adopted DNG as its recommended format for archiving raw files, deeming it “a secure openly documented and forward compatible format for image archiving.”
Adobe provides a free DNG converter that can translate many proprietary formats into DNG. They regularly update it to support new cameras in parallel with Adobe Camera Raw, the raw converter plug-in for both Photoshop and Lightroom. The DNG converter can optionally embed the original raw files into DNG files to retain compatibility with manufacturers’ raw converters that can’t read DNG.
The DNG converter is also Adobe’s magnanimous gift to Photoshop users who have, for whatever reason, fallen behind on the upgrade treadmill. The Adobe Camera Raw plug-in for each Photoshop version is intentionally incompatible with previous Photoshop versions. And Adobe stops updating old versions of Adobe Camera Raw once a new version of Photoshop is released. So if you buy a new camera you’ll probably need to upgrade Photoshop, since only the current version will be able to read the new camera’s raw files.
But if your budget can’t accommodate both the new camera and a Photoshop upgrade, Adobe provides the alternative of using their free converter to translate your new camera’s raw files to DNG, which every Photoshop version since CS can read (the converter even installs the necessary camera profiles). Adobe thus gives you the choice of either paying them $200 for a Photoshop upgrade, or climbing aboard the DNG bandwagon.
A few high-end (Hasselblad, Leica) and minor (Ricoh, Samsung) camera manufacturers have already adopted DNG, possibly with some “assistance” from Adobe. Coming late to the digital table, they stand to benefit the most from using Adobe’s specification rather than developing their own raw formats. Whether the major players ever adopt it may depend on their marketing strategy. Canon currently gives away their Digital Photo Professional software to anyone who buys their cameras, so they would seem to have little to lose from switching to DNG. But Nikon sells their advanced Capture NX 2 raw converter as an extra-cost accessory. They would seem to have no incentive to give up their proprietary format and the “ancillary revenue” it provides. (Nikon cameras come with the snapshooter-oriented Nikon View NX software, which provides rudimentary raw conversion.)
Nikon received many complaints when they released new digital SLRs in 2005 that encrypted the white balance information in their raw files. Nikon issued a defiant press release defending the encryption as essential for the “preservation of [Nikon’s] unique technology.” In response, a “working group of photographers” formed OpenRaw, which promotes making complete documentation available for past, present, and future proprietary raw file formats. They carefully distinguish this initiative from Adobe’s goal of a single universal standard, which they consider impractical. Regardless, Nikon’s avaricious encryption of a minor part of their raw files may have paradoxically increased the demand among photographers for open access to those files. But OpenRaw seems to have ended its active efforts in 2006, possibly because camera manufacturers were not persuaded by their arguments for divulging valuable trade secrets.
A cynic might suspect that Adobe has an ulterior motive behind their apparent altruism in developing and promoting a “public” standard. Adobe Camera Raw is probably the most popular third-party raw conversion software, by virtue of its inclusion in Photoshop, Elements, and Lightroom. The initial version of Adobe Camera Raw 6, released in May 2010 with Photoshop CS5, supported 286 different cameras. Each subsequent update will add more. That means Adobe must incur significant expense continually reverse-engineering or licensing all those proprietary formats.
A single open standard would make things so much easier for them. Yes, it would also make things easier for other developers of third-party raw converters, but that’s not a problem. Adobe can just buy out any competitors that have a promising or threatening product, as they did with Pixmantec and RawShooter. The other benefits, including protecting photographers from obsolescence, are merely selling points. Although Adobe calls the DNG specification “public,” they have patented it. But they generously grant a free license to anyone who wants to develop software or hardware that uses DNG files; it’s the same arrangement they’ve made for their “public” but patented PDF format.
Cynicism aside, DNG does seem a promising technology that could have genuine benefits for photographers, if it ever catches on. I don’t use it, but my Canon Digital Rebel XT and Powershot S90 are very popular cameras that Canon and Adobe seem likely to support for the foreseeable future.
I would consider DNG if I had a collection of raw files from a camera whose manufacturer has gone bankrupt or has left the market (e.g., Konica-Minolta), or whose manufacturer that seems likely to leave the market due to an insignificant market share (e.g., Sigma). It might be more sensible to save the best output of your favorite raw converter as an uncompressed 16-bit TIFF file and/or a 16-bit flat PSD (native Photoshop) file (see A Bestiary of File Formats for more information). Both formats have enormous user bases and enough history to ensure that they will remain readable for any foreseeable future. The American Society of Media Photographers apparently shares that belief, since their “best practice” recommends those formats for archiving finished images.
It’s still too early to tell whether DNG will actually provide a long-term solution to raw-file obsolescence. If any company has the market clout to convince (or bully) camera manufacturers into adopting a standard, it’s Adobe— unless Microsoft decides to enter that arena with a proprietary rights-managed format. But if DNG fails to reach critical mass, it could disappear faster than the formats it’s meant to replace.
Sidebar: My Canon S90 illustrates one shortcoming of DNG as an archival format. Like many compact cameras, the S90 (along with its successor, the S95) has a lens that is not fully corrected for distortion and chromatic aberration. It relies on software to make those corrections, which in the past would have been made with additional lens elements. This approach significantly reduces the size, weight, and cost of the camera while still providing excellent image quality after correction.
The camera’s firmware automatically applies the corrections when it creates JPEG files, but not raw files. (The S90 and S95 are among the very few compact cameras that can create raw files.) Canon’s Digital Photo Professional raw converter applies the corrections when it processes raw files in the camera’s native format. Conversion to DNG would preclude the use of Canon’s converter, which can’t read DNG. Choosing the option in Adobe’s DNG converter that embeds the complete original raw file in the DNG gets around that limitation, although it doubles the size of the file. Adobe Camera Raw 6 (which works with Photoshop CS5) can make the lens corrections for native raw files as well as those converted to DNG, since it uses separate (reverse-engineered) lens profiles included with the plug-in or available for download.
However, if I wanted to use the Adobe DNG converter to process my S90 raw files in Photoshop CS3 (for example), its version of Adobe Camera Raw would not be able to make the lens corrections. Automatic profile-based lens correction first appeared in Adobe Camera Raw 6.1, which works only with CS5. Any other third-party raw converter that can read DNG files would have the same problem, unless it had its own lens profiling and correction capability. Similarly, if future versions of Digital Photo Professional and Adobe Camera Raw dropped support for the long-obsolete S90, DNG conversion might preserve a readable image file (assuming, of course, that the DNG format remains supported). But the distortions and aberrations would have to be corrected manually.
DNG might be a first step toward future-proof raw files. But as this example demonstrates, it’s not a complete solution. For now, the best approach might be to save the save the best output of your favorite raw converter as a 16-bit uncompressed TIFF or flat PSD file.
Following is a brief discussion and comparison of seven file formats for photography and graphics. Don’t look for a complete or scholarly treatise. It’s merely my take on what they’re for, where they come from, their future prospects, and particularly how I use them.
TIFF (“Tagged Image File Format”) is the grand-daddy of image file formats, originating in the mid-1980s. Designed from the outset to be flexible and extensible, it has many mutually-incompatible variants for specialized uses ranging from fax machines to desktop publishing. But in its simplest form of a single 24-bit or 48-bit color image without compression, TIFF is probably the most durable, compatible, and widely supported of the current file formats that can store full image data. With the notable exception of Web browsers, nearly all software that displays or edits images can read it. That compatibility comes at the price of large file size: each pixel of a color image requires three (or possibly six) bytes. That makes it unattractive for Web use, which is why browsers don’t support it. Some software allows compression, but the resulting files aren’t much smaller and may be unreadable in other programs. The compression option in Photoshop actually increases the size of 48-bit TIFF files!
TIFF can store the metadata that digital cameras and some image editing programs create to record date, time, exposure, color space, owner, copyright, and other information. It also accommodates multiple channels (e.g., “alpha” channels for storing selections in image editors, infrared cleaning data from scanners, and the usual red, blue, and green). Photoshop’s implementation can store layers, making TIFF nearly interchangeable with its native PSD format.
I use TIFF for storing full-resolution finished, non-sharpened “master” images. From them I make everything else: the smaller versions for this Web site, prints, and any other products purchasers might want to license. I believe these simple uncompressed TIFF files are very likely to remain supported in future software, which makes them suitable for archival storage of finished images.
JPEG (“jay-peg,” named for the Joint Photographic Experts Group that developed the standard for it in 1992) is ubiquitous on the Web and in digital cameras for full-color 24-bit photographs. Every graphical browser supports it, and nearly all software that displays or edits images can read it. It uses “lossy” compression that attains very compact file sizes by cleverly exploiting the quirks and limitations of human vision. It discards a significant amount of data that the human eye and brain won’t miss. Most software and cameras that create JPEG files can vary the compression to balance quality against file size. Too much compression means smeared color, lost detail, and artifacts. JPEG is limited to 8 bits per channel. It can store metadata but not alpha channels or layers.
JPEG with low compression (“maximum” or “highest” quality) is a good choice for storing final versions of images that don’t require further editing. When viewed or printed, such files are essentially indistinguishable from much larger uncompressed or losslessly-compressed files. But the lossy compression means that repeatedly opening and re-writing a JPEG file degrades the image. It’s rather like the way photocopies of photocopies get increasingly fuzzy. The lost data may also limit and complicate adjustments in an image editor, which can be a problem with digital camera JPEG files.
Because JPEG is so pervasive, it’s another likely candidate for future-proof archival storage, subject to its inherent limitations. I use JPEG for the small images on this Web site, and for delivering high-resolution licensed images to purchasers by download or e-mail (delivery in other formats may require mailing a CD, as the file size could be too large for Internet delivery).
GIF (“Graphics Interchange Format,” pronounced with either a hard or soft g) is nearly as old as TIFF. Every graphical browser supports it, and nearly all software that displays or edits images can read it. It keeps file sizes small through “lossless” compression— when you uncompress the data you get back exactly what you started with. GIF is not useful for photographs because it allows a maximum of 256 colors. But it is useful for text or drawings, since lossless compression doesn’t create the artifacts and smearing that JPEG often produces with sharp, high-contrast edges. GIF supports “transparency,” which means you can designate one of the 256 colors as “invisible” so the background shows through it. That way, lettering, logos, or other images can appear to “float” on the page regardless of the color or background. I use GIF files (with transparent backgrounds) for the graphical titles at the top of the Travel Photo Essay pages on this Web site. Another common use of GIF is for animated images, since the GIF standard allows multiple images in a single file that Web browsers can display in rapid succession.
PNG (“Portable Network Graphics,” pronounced ping or P-N-G) originated as a “patent-free” replacement for GIF. CompuServe developed GIF in 1987, unaware that Unisys owned a patent on the “LZW” algorithm chosen for its lossless compression. Unisys slept while GIF became popular, first on computer bulletin boards and then on the nascent Web. In 1994 they woke up and put their lawyers to work, ordering developers of commercial software that creates GIF files to start paying royalties. Five years later, perhaps out of a greedy desire to squeeze everything they could out of their patent before it expired in 2003, Unisys announced that developers and even individual users of noncommercial and free GIF software now owed them royalties.
PNG was the open-source community’s reaction to the awakened Unisys. It uses a public-domain lossless compression algorithm and supports transparency like GIF. But it improves on GIF by supporting full 24-bit color, which makes it suitable for photographs. Because the compression is lossless, file sizes are larger than JPEG. Line art and graphics with 256 or fewer colors, for which GIF is most commonly used, produces file sizes comparable to GIF.
PNG hasn’t supplanted GIF because Microsoft chose not to support it fully in their Monopoly Standard browser. Microsoft Internet Explorer versions 6 and earlier— which some people continue to use despite Microsoft’s efforts to kill version 6— do not support the “alpha channel transparency” used for color images with transparent backgrounds; all IE versions display incorrect colors in some PNG files (Mozilla/Firefox, Safari, and Opera fully support PNG). I learned that the hard way when I tried to replace the graphical titles on my Travel Photo Essay pages with new PNG versions. To accommodate the majority of my visitors, I had to replace the new headers with GIF versions that don’t look as good. Microsoft Internet Explorer 7 finally corrected the transparency problem but not the color problem. Better late than never, I suppose. And better to fix half the problem then none.
PNG has an inherent flaw that makes it useless for modern digital photography. The PNG standard makes inadequate provisions for metadata. While it does allow a few text fields, there’s no standard way to store the exposure data and camera information (EXIF) that nearly all digital cameras record in their raw and JPEG files. And more significantly, there’s no standard way to embed the color space and profile information that’s essential for a color-managed workflow.
Adobe’s PNG plug-in for Photoshop extends the format to 48 bits, but it doesn’t save even the minimal metadata allowed in the PNG standard. fnord software offers SuperPNG, a freeware replacement for Adobe’s plug-in. SuperPNG can store and read full EXIF metadata, including color profiles. It also works with 48-bit images. It seems to work as advertised, but I don’t know if any other software can read the metadata. SuperPNG produces somewhat smaller 48-bit files than Adobe’s plug-in, but JPEG 2000’s lossless compression does even better. SuperPNG was last updated in April 2007, and does not work with 64-bit versions of Photoshop.
JPEG 2000 was supposed to be a 21st century replacement for the 1992-vintage JPEG. Its superior lossy compression technology can produce fewer and less noticeable artifacts with smaller file sizes. It also includes a very effective lossless compression option. Despite its advantages JPEG 2000 has gone nowhere, apart from a few specialized applications outside the photographic or consumer realms. A decade after its release, no Web browser supports it natively. (Browsers can display JPEG 2000 files using plug-ins and extensions, but designers of Web sites can’t count on users having those installed and properly configured.) Part of the problem may be that the “legacy” JPEG format has proved entirely adequate in very extensive use, so nobody is clamoring for a replacement.
Another difficulty is that image compression algorithms are a minefield of overlapping and broadly-drafted patents, in part due to the understaffed and inept United States Patent and Trademark Office. The committee that created JPEG 2000 claims to have obtained all the necessary rights from the owners of the various patents covering its underlying technologies. They’ve made “a strong goal” of allowing anyone to implement the basic functionality without royalties. But even with licenses, those patents effectively preclude the adoption of JPEG 2000 within the influential open-source community. And there remains a pervasive fear that someone might suddenly emerge with an unnoticed and possibly dubious “submarine” patent, and attempt to shake down developers the way Unisys did with GIF and Forgent Networks did with JPEG. Forgent’s patent was eventually invalidated, but not before they extorted over $90 million from 30 companies and sued 31 others.
Photoshop CS2 and CS3 support JPEG 2000 through a plug-in buried in an “optional plug-ins” directory on the installation CD or DVD. You have to know about it and install it manually. Its lossless mode provides the most space-efficient way to store 48-bit images, including metadata and alpha channels. Reading files is very slow. Writing files is excruciatingly slow, as the plug-in makes multiple passes through the file to optimize lossy compression even if you’ve selected lossless compression. A “fast mode” check-box option disables this optimization and makes file writing somewhat faster, but it often mysteriously un-checks itself and forces you to wait for the useless optimization before you can reset that option.
Adobe seemed to have signed JPEG 2000’s death warrant with the release of Photoshop CS4. No longer included on the installation DVD, the plug-in was relegated to a boneyard collection of “legacy optional plug-ins” available from Adobe’s download site for Windows and Macintosh. But they resurrected JPEG 2000 in CS5, which installs a somewhat improved version of the plug-in as a standard part of Photoshop. This change of direction suggests that at least some of Adobe’s customers are finally becoming interested in the format, though I suspect it’s mainly the ones who buy the Extended version of Photoshop for specialized scientific, technical, and video applications.
Despite the significant space saving, I can’t recommend JPEG 2000 for archival image storage. Few graphics programs can read it, and as far as I know only Adobe’s plug-in can read the lossless 48-bit files it writes in Photoshop. And despite its apparent revival in CS5, it’s impossible to predict the future of JPEG 2000. But I use it for saving 48-bit “working” files I create in the process of editing images. It saves significant space when I archive my projects to DVD, and Adobe provides no real alternative.
JPEG XR (also known as HD Photo) is the newest animal in the file format zoo. It’s Microsoft’s answer to JPEG 2000, and offers very similar features and performance. Microsoft originally called this proprietary format Windows Media Photo. But when they decided they wanted to monopolize digital imaging the way they’ve monopolized office applications, they renamed it HD Photo. (“HD” seems to be the faddish name currently applied to new digital technology whose benefits may not be readily apparent to many who are supposed to adopt it, as in HDTV and HD Digital Radio.)
Microsoft started promoting HD Photo at the beginning of 2007. They appointed a “Director of Digital Imaging Evangelism” to spread the gospel. Yes, that’s his actual job title! As proof that “it’s not just for Windows anymore,” they’ve supposedly promised to make the HD Photo specification available for free. They also suggested that they intend to refrain from suing developers who implement the patented technology on non-Microsoft or open-source platforms. But Microsoft’s lawyers have yet to memorialize this “intent” in any form that might actually clear the way for such development.
While Microsoft’s evangelists haven’t yet saved the souls of many users or software developers, they have obtained (or purchased?) the endorsement of four international standards bodies. The Joint Photographic Experts Group has given it the JPEG cachet as as “JPEG XR” (XR stands for “extended range”); the International Organization for Standardization and the International Electrotechnical Commission have jointly designated it as “ISO/IEC 29199-2;” and the International Telecommunications Union has approved it as “ITU Recommendation T.832.” Microsoft has offered royalty-free licensing of the relevant patents as an additional incentive. Microsoft’s control of those patents may actually work in JPEG XR’s favor, as their armada of lawyers stands ready to torpedo anyone who threatens a “submarine” attack.
Despite the evangelism and the prestigious international recognition, JPEG XR doesn’t seem have garnered much support outside of Microsoft. Their Expression Studio Web design suite can create the files, and the media viewers included with Vista and Windows 7 can read them. Version 9 of Internet Explorer also includes native support. The only non-Microsoft applications I’m aware of are the freeware viewer/editors IrfanView and Paint.NET, which offer optional plug-ins that can read JPEG XR files. For a while Microsoft offered a free Photoshop plug-in, but that has now quietly and inexplicably disappeared from their download site.
Despite Microsoft’s marketing clout, JPEG XR— under any of its various designations— is clearly facing the same uphill battle JPEG 2000 had, to convince users that it offers significant and valuable advantages over established and entirely adequate formats. The evangelists also have the particular challenge of selling HD Photo to an inherently skeptical community. Many photographers and graphic designers are loyal Apple Macintosh users who are inherently resistant to any proprietary Microsoft “solution.” And other potential users could have valid concerns about Microsoft’s intentions. I downloaded Microsoft’s Photoshop plug-in when it was available. But when the installer presented me with an “End User License Agreement” full of legalese, I decided that JPEG XR probably isn’t worth spending the time to read and “agree” to it. If that’s an indication of how Microsoft plans to “evangelize” HD Photo, I can see why it hasn’t gone anywhere.
At least for now, JPEG XR seems to be a solution in search of a problem— or at least a problem other than Microsoft’s desire to control digital imaging. (Microsoft now seems to have a predilection for Himalayan uphill battles. Besides JPEG XR, they have ambitions for their XPS and Silverlight to respectively supplant Adobe’s ubiquitous PDF and Flash. If these new formats indeed offer users any compelling advantages over the entrenched de facto standards, Microsoft’s evangelists have yet to elucidate any of them.) Still, if any company can “evangelize” their proprietary format into a monopoly standard, it’s Microsoft. So stay tuned. But I’m not holding my breath.
Finally, PSD is Adobe Photoshop’s native file format. As you’d expect, it supports everything Photoshop can produce, including layers. To my knowledge, TIFF is the only other format that supports layers, although non-Adobe software may have problems reading files with layers. The main disadvantage of PSD is a file size as extravagant as TIFF, multiplied by any saved layers. The only compression it allows is the nearly worthless “run-length encoding” that compresses some sequences of repeated bytes.
PSD is another possible candidate for future-proof archival storage because of the long-standing thorough dominance of Adobe and Photoshop in the graphics market. Some non-Adobe products like Paint Shop Pro can read it. I use when I need to preserve layers in working files, although TIFF can do that too.