Friday, 11 February 2011

Embedded in RAW

One of the most common debates (ignoring brand wars) in the hobbyist/amateur camera world is whether one should shoot RAW or jpeg (or RAW/jpeg with varying degrees of jpeg quality). For me the decision is easy: shoot RAW.

The main reason people talk about shooting jpeg, ignoring the storage capacity issue, is that there's no time required/desired to post process the RAW files simply to generate simple review images.

Let's start by stating one thing:
a RAW file includes, embedded in the metadata, a full size preview jpeg
This statement is true for both Nikon NEFs and Canon CR2s and in fact, there can be several sizes embedded. For unmodified/straight out of camera RAW files, the embedded jpegs tend to be of basic quality and not the best jpeg quality that can be generated by the camera.

However, the embedded preview jpeg contained in Nikon NEFs that have been modified/updated by CNX2 are of higher jpeg quality. A Nikon D300 generates fine jpegs in the ~3-4MB range and basic jpegs in the ~1MB range depending on the scene/image. Previews extracted from an unmodified NEF are in the ~1MB range but the preview image from the same NEF following simple WB correction in CNX2 are in the 4-5MB range.

The only task is to extract the preview images from the RAW files to provide the 'quick/mass review' which can be done simply with readily available tools.

Preview Image Colour Space Notes

RAW files are recorded without a relation to any colour space (sRGB/aRGB etc) - however:
the embedded preview jpegs are associated with a colour space
The implication is that any preview image extracted may need colour space conversion (to sRGB for instance) if the camera body is configured to be in aRGB mode.

Nikon NEFs' preview images are also updated when the RAW file is edited in CNX2 - remember that CNX2 writes back updates to the NEF file itself, which differs from other Nikon RAW editors which have to store the edits are steps in a separate database etc. The consequence of updating NEFs in CNX2 means that the preview image will be in the colour space as specified at the time of edit.

CNX2 provides the option to embed the ICC profile in the NEF metadata when saving postprocessing - this option is always enabled in my setup. The ICC profile is stored in the IFD0 container with the standard Exif ICC tag. However, if a jpeg is generated from the NEF in CNX2, the jpeg will store the ICC in the APP2 container.


There are a number of simple tools to extract RAW files' embedded preview images - exiftool can be scripted to extract to do this. Aside from this use case, exiftool is a fantastic tool that allows for examination of an image's metadata, including indepth binary analysis of EXIF containers.

But there are 2 preferred tools for extracting previews from our RAW files: Exiv2 or imgprextr.

Note that the resulting extracted jpegs will be in the colour space as specified by the camera at the time of capture or in the colour space saved/specified by the RAW editor which may not be desirable: for example, AdobeRGB jpegs intended for web are not always displayed correctly by browsers.


exiv2 is a powerful EXIF manipulation cmd line utility, available as an UNIX tool or 32bit MS Windows executable, which can also be used to extract the embedded preview images. To query the availability of preview within a RAW file:
$ exiv2 -pp DSC_xxx.NEF
Preview 1: image/tiff, 320x212 pixels, 203520 bytes
Preview 2: image/jpeg, 4288x2848 pixels, 3269405 bytes
$ exiv2 -ep2 DSC_xxx.NEF
This returns preview image identifiers and sizes and shown above, we extract the largest preview (id #2) using the -ep2 switch.


Further to the tools noted above, another powerful set of tools are provided by ImageMagick, particularly the convert and identify utilities. IM provide source code (for the UNIX users) and binary distributions for MS Windows and these tools will you to perform any colour space conversion (amongst many many other tasks).

If a colour space conversion is required, it is a pre-requisite that the source and destination colour profiles are known. To perform the colour space conversion:
convert  \
-profile aRGB.icc  DSC_xxx-aRGB.jpg \
-profile sRGB.icc  DSC_xxx-sRGB.jpg
The example peforms an AdobeRGB to sRGB conversion.

Whilst IM can provide conversions from RAW formats to other image formats, IM (at least 6.6.7-7) uses a ufraw component (in turn, derived from dcraw) to the RAW conversion which I have found to generate dark images when using default command line options.

Furthermore, RAW conversions are CPU intensive operations so it is simply more efficient and quicker (and at this moment, more accurate) using exiv2 to extract preview images.

One last note on IM: whilst it can perform numerous image format conversions, ImageMagick does not appear to be able to extract all embedded preview images - at best it appears to be able to extract the smallest preview image only.


Above and beyond the available utilities, they provide APIs that can be used, such as the exiv2 C++ API, to provide programmatic access to the metadata and the preview images.

One issue that we have with exiv2 is that whilst it can extract the preview image, it takes no responsibility for colour spaces of the preview - it will be left as found. As noted previously on this post, jpegs may be more suitable for mass consumption (web: flickr/facebook/google+/...) if they were converted to sRGB.

The problem with post RAW preview image extract conversions is that we will need to manually determine the colour space of the preview image and then have the target colour space file to perform the conversion. As you can imagine, this is a cumbersome, tedious and error prone set of steps that are better handled at the stage of preview image extraction and the reason I wrote imgprextr.

imgprextr is part of my other set of utils source code.

Usage is simple:
imgprextr -p jpeg -c srgb  *.NEF

The example above will place the extracted preview images in the location specified by the -p flag, attempting to create the directory if it doesn't exist. The -c srgb flag requests a sRGB colour space conversion if required - any other parameter will be treated as the location of an ICC profile for the target colour space. Without the -c flag the tool will simply extract the preview image as it found them.

The Win32 command line executable for imgprextr is available here.

No comments:

Post a Comment