[Feature request] Use Non-Native Channels in Trichrome Scans

[matthoosier]

NegPy currently throws away quite a bit of information during a narrowband trichrome scan. Each exposure is demosaiced into the usual R, G, and B pixel planes, but when the three exposures are merged, only the native channel from each exposure is retained: R from the red exposure, G from the green exposure, and B from the blue exposure. The non-native channels are discarded on the assumption that they are mostly attenuated crosstalk and therefore not particularly useful.

But the CFA sensitivity curves of real cameras overlap substantially. In some cameras, the G channel can respond nearly as strongly as the B channel to narrowband blue light, for example. And even where the response is much weaker, there can still be a considerable amount of spatially resolved signal sitting well above the noise floor. More importantly, in a trichrome scan we know exactly where that signal came from. If the illumination is narrowband red, then whatever the green and blue CFA cells record during that exposure came from the red illumination. It isn't an unknown mixture of scene colors. The problem is primarily that those cells are being underexposed relative to the native channel.

This suggests a fairly straightforward form of exposure bracketing. Instead of taking one exposure for each narrowband primary, take three: one exposure calibrated for each of the three CFA channels. Under red illumination, for example, the first exposure would be the normal red exposure, with the R channel brought into the useful part of the histogram. A second, longer exposure would be chosen to bring the G response into that range, and a third would do the same for B. The R channel would of course clip in the longer exposures, but that reading can simply be discarded because we already have the properly exposed R measurement from the first exposure. The important point is that the shutter lengths need to be calibrated independently, since the CFA responses are different for each channel.

The calibration wizard could therefore be extended to measure the response of all three CFA channels to each narrowband primary and determine the appropriate shutter length for each of the nine combinations. The scanning automation would then take the resulting nine exposures as one logical trichrome scan: three under red illumination, three under green, and three under blue. Each exposure would need to retain its illumination color and target CFA channel so that the longer exposures are not confused with additional ordinary triplet exposures.

The merge could then make use of the additional data in the linear RAW domain. Each measurement would be normalized for its exposure time, saturated values would be rejected, and where more than one exposure provides a usable measurement of a particular channel, the measurements could be combined according to their relative signal-to-noise. The final output would still be the same three RGB planes expected by the rest of NegPy. The difference is that instead of treating the non-native CFA response as something to throw away, the scanner would deliberately expose for it and use it as additional spatial information. In other words, the three narrowband exposures become three independent exposure brackets, allowing the CFA's spectral overlap to become useful signal rather than simply crosstalk.

[matthoosier]

I should have said above that this is more of a feature proposal than a request. I could probably contribute the implementation.

But I'm curious what the reaction to the idea is. The main idea is to exploit repeated exposures of different lengths to get direct sensor readings at every Bayer sensor site under every LED narrowband primary. So you would avoid the interpolating that comes with demosaicing entirely.

I suppose some camera bodies just wouldn't have enough crosstalk for this to give any useful sensor data for some colors. In that case, the calibration wizard could just issue an error saying that no length of shutter under e.g. red narrowband yields photon reception in the Bayer cells behind the blue CFA filters that are above the noise floor.

Reactions, @marcinz606 ?

[thetalkingdrum]

Interesting idea from a technical perspective, but have you done any real world testing on this? Feels like there are several things that could make this less ideal than it seems.