Match Reference and Macbeth Dynamic Range

[tomczak]

Why would setting the Dmin=0 and Dmax=3.7 in Match Reference on the processed RAW image be the right settings to get the set of Colour curves so that the 6 gray squares at the bottom are mapped to their assumed values for the gamma of the working colour space? What's the mathematics of it?

It seems that the deltaD between the darkest and whitest square is ~1.45 (providing that the exposure is such that they are all within the image range). Most often, neither the white square, nor the black one are the WP and BP of the image. Experimenting with the card and RAW, I often get shaddows or highlights clipped because of the Colour Curves step in Match Reference, no matter what I do. I thought that the theoretical black square (which is at about normalized pixel level of 21%) and the theoretical white square (which is at some 95%), given the working space gamma=2.2, could correct the exposure while giving enough headroom in both tails, for both darker and brighter pixels.

I keep failing, probably because I don't understand enough. Could someone please help?

[jsachs]

The dynamic range of the Macbeth target is limited - in addition, the dynamic range of the camera image is affected by lighting, exposure and the dynamic range of the sensor. The settings in Match Reference let you adjust for all this by tweaking the black and white density settings as necessary to keep the dynamic range of the result image from maxing out at either end. Due to all the variables there is no way to do this fully automatically.

[tomczak]

Many thanks. I did a few more 'reverse engineering' trials with Match Reference and it's curves, trying to figure out how to use it with grace with RAW files. It seems that:

- typically, the photographed Macbeth dynamic range is well within the processed RAW range (under different lighting, exposure etc., as long as it's not blown at either end). For instance: the white/black patches from the target could fall as say 90/30% gray in the image. Picture is flat and needs contrast stretching first, which the first step of Match Ref. could do, I figured.

- When the Match Reference automatically sets the Colour Curves, it tries to map the 6 gray squares from the colour checker to their theoretical RGB values given the gamma of the working colour space (while removing casts). At gamma 2.2, these normalized pixel values are around: 0.21, 0.33, 0.48, 0.63, 0.79 and 0.95.

- Each RGB curve created, has 8 control points: 6 points for mapping each of the original gray patch value to each of the theoretical ones, plus two endpoints.

- It's the endpoints (i.e. points on the curve that don't correspond to any of the colourchecker gray patches, but determine what's ultimately clipped in both tails), I think, that confuse me: why are they 'linear'? extensions of the actual curve, rather then sloping gently to the (0,0) and (1,1) corners? This is what I think causes all the clipping of processed RAW in practice, and I can't find a way of controlling it in Match Reference.

- if the Black/White density sliders in Match Ref. are set to 3.7/0, this is what happens, more or less. Limiting the dynamic range further with B/W density sliders will make the slopes of the curves steeper (clipping the ends even more).

[tomczak]

There is another aspect of Match Reference that I don't understand.

I have two images of the Macbeth chart: one a little bright, the other a little darker, but none of the patches is blown or blocked. For the brighter one, the values of the patches happen to be v. close to their theoretical values in RGB, gamma=2.2, so in the first step the RGB colour curves are mostly diagonal. For the darker image, the curves step, more or less, shifts the diagonal up (i.e. it shifts the histograms to the right, without stretching them, thus brightening the image - but the original RAW conversion gamma and WB are about the same for both).

The net result is that after the curves, the gray patches of both images match, but the saturation of their colour patches is quite different: the brightened underexposed image has its colours washed out much more than the image that didn't need the brightening. The Colour Correction step differs as a consequence, not that much in the vectors directions, but their magnitude.

I was hoping that the colour curves step would equate the white point, gamma and contrast of the Macbeth Chart at different exposures, and then colours can map the straight RAW converted image, brought to proper exposure with the curves, to the theoretical Macbeth value. They shoud be similar if the curves brought the image to some common exposure point, but they are not.

I suspect that brightening each of the RGB channels separately when adjusting the Macbeth grays, lightens the colour patches and decrease their saturation. Is it theoretically possible to do curves in HSV space which, if anything, would increase the saturation when shifting the histogram right?

What else can be done to take the unequal exposures out of the equation, when using Macbeth as Camera/RAW converter calibration of sorts?

[jsachs]

What exactly are you trying to use Match Reference to accomplish?

[tomczak]

I'm trying to use Match reference in two separate projects:

A) to calibrate the camera+RAW with what I have (i.e. camera, macbeth and PWP). I would like to bring the RAW colours to a standard, from which I could edit it more.
B) to use it as intended i.e. to correct images where colour checker is part of the image or is taken separately but under the same conditions as the image to be corrected.

I seem to be having problems with both, for a different reason. I think my problem can be isolated to the .cc step. It's best to show it by example.

Image 1 happens to be exposed so that the Macbeth gray patches values are about the same as Match reference wants them to be (i.e. the .cc step makes little difference). Image 2 is a little darker (but well within its dynamic range), which causes Match Reference to lighten it in .cc step by shifting the histogram to the right. Both images have little overall colour cast or gamma irregularities. The working colour space has gamma 2.2.

The Macbeth images (i.e. Image 1c and 2c) end up being similar and closely match the theoretical Macbeth values, when the Match Reference is done. But with real images there are 2 problems which I have trouble solving:

In project A): the .colorcorrect step depends on the output of the .cc step. If the RAW exposure is such that the .cc step does little, the .colorcorrect step works fine. But if brightness needs to be adjusted (e.g. increased), the .cc step, operating on RGB space, washes down the colours and forces the .colorcorrect step to increase saturation. I don't know how to come up with a standard correction(s) that would neutralize the colours specific to the camera/RAW processor to something standard, so that I can bring them to where I want them to from there.

In project B): the 'black' and the 'white' patches on Macbeth have theoretical, 2.2 gamma corrected brightness of some 21% and 95%. That's where the .cc step wants to put them. Since they are the darkest and brightest objects on the Macbeth, the chart is corrected properly. But usually, the scene contains object darker and lighter than that and they will be clipped by the .cc step of the Match Reference (in the example 2 here, anything lighter than 95% is blown). The black and white density sliders don't prevent it. One reason for clipping, I figured, could be how the endpoints of the .cc curves are mapped.

[tomczak]

I forgot to ask: is the .cc step done internally in RGB space and, if it is, does it have to? Would HSV work any better?

[tomczak]

As I'm trying to return to this project, a naive question: would increasing the maximum value of black density allowed by Match Reference slider be a quick fix that could be used to limit clipping of shadows and highlights (those outside the range of the brightest and darkest patch of Macbeth Chart) by the Curves step of the transformation?

Or do I have it all completely wrong...?

[jsachs]

The best way I can think of to eliminate clipping would be to crop the Macbeth image out of the total image and then use Brightness Curve in HSV mode to stretch the dynamic range of the reference image so that it runs all the way from 0 to 100%, before using the Match Reference transformation.

[tomczak]

I'm struggling with details in my experiments with Macbeth, and I just want to check my understanding of fundamentals of Match reference, which I thought I understood, but I'm not sure any longer.

In theory, with monitor calibrated, if I take a properly exposed picture of Macbeth colour chart illuminated with a light with colour temp. of 6500K, develop the RAW in some standard way, and then subject it to Match Reference using a working colour space with WP=6500K (such as sRGB or AdobeRGB), I should end up with on-screen Macbeth chart looking as if it was illuminated with a 6500K light (i.e. the Match Reference should negate colour shifts introduced by this particular camera and RAW processing). Is that true?