I'm attempting to use the stencil buffer as a clipper for my UI system, but I'm having trouble debugging a problem I'm running in to.
This is what I'm doing: A widget can pass a rectangle to the the stencil clipper functions, which will increment the stencil buffer values that it covers. Then it will draw its children, which will only get drawn in the stencilled area (so that if they extend outside they'll be clipped). After a widget is done drawing its children, it pops that rectangle from the stack and in the process decrements the values in the stencil buffer that it has previously incremented.
The slightly simplified code is below:
static void drawStencil(Rect& rect, unsigned int ref)
{
// Save previous values of the color and depth masks
GLboolean colorMask[4];
GLboolean depthMask;
glGetBooleanv(GL_COLOR_WRITEMASK, colorMask);
glGetBooleanv(GL_DEPTH_WRITEMASK, &depthMask);
// Turn off drawing
glColorMask(0, 0, 0, 0);
glDepthMask(0);
// Draw vertices here
...
// Turn everything back on
glColorMask(colorMask[0], colorMask[1], colorMask[2], colorMask[3]);
glDepthMask(depthMask);
// Only render pixels in areas where the stencil buffer value == ref
glStencilFunc(GL_EQUAL, ref, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}
void pushScissor(Rect rect)
{
// increment things only at the current stencil stack level
glStencilFunc(GL_EQUAL, s_scissorStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
s_scissorStack.push_back(rect);
drawStencil(rect, states, s_ScissorStack.size());
}
void popScissor()
{
// undo what was done in the previous push,
// decrement things only at the current stencil stack level
glStencilFunc(GL_EQUAL, s_scissorStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);
Rect rect = s_scissorStack.back();
s_scissorStack.pop_back();
drawStencil(rect, states, s_scissorStack.size());
}
And this is how it's being used by the Widgets
if (m_clip)
pushScissor(m_rect);
drawInternal(target, states);
for (auto child : m_children)
target.draw(*child, states);
if (m_clip)
popScissor();
This is the result of the above code:
There are two things on the screen, a giant test button, and a window with some buttons and text areas on it. The text area scroll box is set to clip its children (so that the text doesn't extend outside the scroll box). The button is drawn after the window and should be on top of it completely. However, for some reason the text area is appearing on top of the button. The only reason I can think of that this would happen is if the stencil values were not getting decremented in the pop, and when it comes time to render the button, since those pixels don't have the right stencil value it doesn't draw over. But I can't figure out whats wrong with my code that would cause that to happen.
