PWP 7.0.8-32 -- Bad Pixels - Have you tried it?

[den]

No problems here, just encouraging investigation and discussion...

If you have not explored Transformation/Special Effects/Bad Pixels, do so. It can be very powerful, especially for older camera sensors and if you regularly take timed exposures that exceed two seconds... ...although I have had occasions with my camera, where one or more of the more brighter Stuck pixels will show up in an image's shadow tone range with exposure times well below two seconds...

Use the F1 Help [when the "Find Bad..." dialog is 'active' in the work space] as an initial guide to get started...

As a substitute for the 'white' image, one would could simply use a New 100% white [R=G=B=100% or 255] color image based upon the raw converted 'black' lens cap image...

...for the "Find Bad..." transform to work, both the 'white' and 'black' images must be at least 24 bit color; have a 1:1 pixel to pixel registration including overall dimensions; and conversions from their respective raw files.

Illustrations...

The following scene was taken with a 10 second timed exposure using a tripod and ambient room incandescent lighting with outdoor lighting behind lightly green hued drawn curtains... ...a white 3.5x5 inch card in front of the potted plant was used as a mid-tone Color Balance neutral gray reference...

...the indicated yellow 1:1 Zoom factor 267x400 pixel image area is further illustrated: left, un-corrected bad pixels and right, PWP Raw Dialog S&N tab Average "*.badpixel" fixed...

...Stuck pixels need not be confined to the bright specks but any one or more of the R, G, and/or B channels where Stuck pixels have less than a 100% Value. These can give false colors compared to their neighboring pixels... ...shown at a 2:1 Zoom factor...


...den...

[tomczak]

Would detecting bad pixels potentially detect sensor dust as well? I haven't tried it yet, but since it uses both black and white images I imagine that it could, but then I've seen dusts specs on dirty matrix, that covered several neighboring pixels, and was not v. sharply defined at that (parts were semi-transparent). How will Bad Pixels react to it? Have anybody tried it?

[den]

Would detecting bad pixels potentially detect sensor dust as well?
How will Bad Pixels react to it?
Have anybody tried it?

No...
There is a non-user threshold setting that Jonathan has set... ...the reaction to 'sensor dust' will most likely result in the message: "Too Many Bad Pixels" as will attempting to use this transform for luma/chroma noise reductions...
No...

Bad pixels are very specific and do not always become visable in every image to the same degree... ...their exact locations can also depend upon interpolation method... ...so as a rule, I will only use a Badpixel file with an raw conversion that uses the same interpolation that the Badpixel file was developed.

The PWP derived Badpixel file can also be used with 'dcraw' using the option: -P "<name>.badpixel.txt" if the txt file is located in the same folder as 'dcraw.exe'.

As a guide, this pdf may be of additional help when using PWP's Badpixel transform... http://www.ncplus.net/~birchbay/tutoria ... ersion.pdf

...den...

[tomczak]

I have a few more questions regarding Bad Pixels:

1) Can the White image really be an artificial white with the same dimensions as the black frame image? What I'm wondering is if there is no sensor-specific information extracted from the white image that can help detecting dead pixels for example?

2) When a white (e.g. grossly overexposed sky) RAW is processed, which settings in RAW matter most: for instance, I forgot to turn off Contrast Expansion and ended up with medium gray instead of white. Does it matter? How about sharpening (I presume that NR should be off)?

3) WB? It makes quite a difference especially in noisy Black image.

4) ISO? Does it matter? Would low ISO be easier on the detection algorithm? Both images should be the same ISO I presume?

5) Do Black and White images have to be RAW and processed with PWP RAW? Would in-camera JPGs do, for example?

[den]

A virtually created white image will not reveal Dead pixels but can be useful in finding the usually more numerous Stuck pixels... ...from say a ten second timed exposure of a capped lens in a dark room [closet, :)].

As Badpixel removals are a feature of DCoffin's 'dcraw', I personally prefer to use 'dcraw.exe' conversions to 16-bit color tiffs for the white and black images as they will be minimally 'edited' given the basic dcraw options suggested in the pdf with no data losses/artifacts that occur with jpeg compression/sharpening/editing...

The pdf suggested HSV ColorCurves will further enhance the visibility and inclusion within PWP7's Badpixel transform threshold, dead and stuck pixels...

To help understand what the Badpixel transform actually does:
(1) A virtual 'white' image with 8 Dead pixels:

8 Dead pixels at edge thirds 400x267image.jpg (7.12 KiB) Viewed 29865 times

(2) A virtual 'black' image with 5 Stuck pixels:

5 Stuck pixels center rectangle 400x267image.jpg (6.92 KiB) Viewed 29859 times

(3) The resulting Badpixel transform screen shot:

screen view 01.jpg (34.9 KiB) Viewed 29870 times

Selecting Yes creates a "8 Dead 5 Stuck.badpixel" file which in text form ["8 Dead 5 Stuck.badpixel.txt"] is and whose coordinate origin 0,0 is the upper left image corner...

133 266 0
266 266 0
0 177 0
133 177 0
267 177 0
399 177 0
200 132 0
0 88 0
133 88 0
267 88 0
399 88 0
133 1 0
266 0 0

The first number in a row is the horizontal coordinate from the left image edge in pixels... ...the second number is the vertical coordinate down from the top image edge in pixels... ...the third number is a needed 'filler'.

In PWP7's Raw Dialog S&N tab, these pixel locations can be either 'Median' or 'Average' blurred with their neighboring pixels during the image's raw conversion... ...effectively implementing a 'Speck Removal' at the designated pixel locations.

Ideally, for an extremly over-exposure white image [sky, white monitor screen, brightly lit sheet of white paper], any pixel location that is less than R=G=B=255 or 100% tone is a Dead pixel and for a timed exposure of a capped lens in a dark room black image, any pixel location that is more than R=G=B=0 or 0% tone is a Stuck pixel...

As you come across 'badpixels' during the post-raw conversion editing that have not been reduced using the initial Badpixel file, you can manually add these pixel locations to the file using the text format... ...generally, once you have a comprehensive file, only occasional updating will/may be necessary...

Currently, for my Canon EOS-350D, the Badpixel file has 11 Dead pixels [near edges] and 1851 Stuck pixels scattered through 3474x2314 pixel images. This file is a compilation of three white/black image sets where the timed exposure for the black image was 2s, 10s, and 30 seconds. The increased raw conversion time using either PWP's RAW Dialog or dcraw.exe when implementing Badpixel reductions is not noticeable.

...den...

[tomczak]

Thanks Dan,

Could you clarify two things for me

1) when taking white and black test images, why would long exposure time matter? Wouldn't stuck and dead pixels show with shorter exposure times? Also, in some cameras exposures longer then .5-1s invoke automatic black frame subtraction NR. Does it matter?

2) How about ISO - should I take the test images at low ISO or would higher ISO make it easier or more difficult for Bad Pixels to make a detection?

I imagine that longer exposure and higher ISO would produce black image with more noise - which I would expect cold mask the stack pixels we are after, or do I have it all wrong? You mentioned that you've taken 3 different long exposures - did one of them work better than others?

Cheers!

[jsachs]

I would recommend the least extreme exposure that makes almost all the pixels black or white - as can be judged from the in-camera histogram. To qualify as stuck or dead, pixels must be a lot brighter or darker than the background so small amounts of noise will not have any effect.

[den]

1) when taking white and black test images, why would long exposure time matter? Wouldn't stuck and dead pixels show with shorter exposure times? Also, in some cameras exposures longer then .5-1s invoke automatic black frame subtraction NR. Does it matter?

Exposure time for dead pixels is not applicable, dead is dead...

Exposure time for stuck pixels is a factor for finding stuck pixels and affected pixels can respond in a non-linear way becoming visible one time but the not the next under nearly the same circumstances... ...[my understanding is that at one time, new camera sensor quality control was such that stuck pixels would not become visible unless exposure time was 2 seconds or more]

2) How about ISO - should I take the test images at low ISO or would higher ISO make it easier or more difficult for Bad Pixels to make a detection?

If you feel that automatic black frame subtraction NR and ISO are factors for your camera, by all means derive specific Badpixel files for those conditions... ...there is no one size fits all...

The only difficulty may come by exceeding the PWP Badpixel transform threshold resulting in "Too Many Bad Pixels".

The suggested pdf HSV ColorCurves will help to established user based thresholds that can be adjusted so that PWP's threshold will not be exceeded and a pixel location table produced.

Dcraw based conversion of the white and black raw image files will help to eliminate the impact of in- and out- of camera non-user adjustable edits.

I imagine that longer exposure and higher ISO would produce black image with more noise - which I would expect cold mask the stack pixels we are after, or do I have it all wrong? You mentioned that you've taken 3 different long exposures - did one of them work better than others?

Longer exposure and high ISO noise really do not matter as PWP's threshold still has to be met in order to produce a badpixel location file...

I chose 3 different black image timed exposures of 2s, 10s, and 30s adjusting thresholds with the pdf HSV ColorCurves so that the Find Bad Pixel transform would execute... ...a majority of the badpixel locations were common for all three images... ...those that were not were combined with the common locations.

If there is further interest, Anyone... ...make available for download, a black camera RAW file and I will produce a Stuck pixel file for that image as long as it is supported by dcraw v9.17... ...and as time permits... ...but I would hope that this thread and pdf provides sufficient guidance to proceed on your own.

I may have gotten a bit carried away with this new PWP7 feature... ...but it is increditably gratifying to "pixel peek" my old Canon EOS-350D's Badpixel fixed RAW converted images!!!!

...den...

[den]

Jonathan, Others...

My apologies... ...I invited discussion and yet unintentionally minimized Jonathan's considered approach to Find-ing Badpixels... ...his transform is more than just a white/black image comparison producing a pixel location table if a threshold is exceeded as implied... ...the considered differential threshold is really the key to this transform so that noise does not overly distort the comparisons and make a "mush" of the image if the resulting file is used.

The approach I was hoping to convey is that there are user available steps that will maximize the number of badpixels found that do not exceed the transform's threshold... ...so that one does not get so frustrated with continuous "Too Many Bad Pixels"...

...den...