I'm playing with some manual (walk-the-pixels) image processing, and I'm recreating the standard "overlay" blend. I'm looking at the "Photoshop math" macros here:

http://www.nathanm.com/photoshop-blending-math/ (See also here for more readable version of Overlay)

Both source images are in fairly standard RGBA (8 bits each) format, as is the destination. When both images are fully opaque (alpha is 1.0), the result is blended correctly as expected:

But if my "blend" layer (the top image) has transparency in it, I'm a little flummoxed as to how to factor that alpha into the blending equation correctly. I expect it to work such that transparent pixels in the blend layer have no effect on the result, opaque pixels in the blend layer do the overlay blend as normal, and semitransparent blend layer pixels have some scaled effect on the result.

Can someone explain to me the blend equations or the concept behind doing this?

Bonus points if you can help me do it such that the resulting image has correctly premultiplied alpha (which only comes into play for pixels that are not opaque in both layers, I think.)

Thanks!

// factor in blendLayerA, (1-blendLayerA) somehow?
resultR = ChannelBlend_Overlay(baseLayerR, blendLayerR); 
resultG = ChannelBlend_Overlay(baseLayerG, blendLayerG);
resultB = ChannelBlend_Overlay(baseLayerB, blendLayerB);
resultA = 1.0; // also, what should this be??