Stencil buffer appears to not be decrementing values correctly

Posted by Alex Ames on Game Development See other posts from Game Development or by Alex Ames
Published on 2012-03-28T07:58:11Z Indexed on 2012/03/28 17:47 UTC
Read the original article Hit count: 302

Filed under:
|

I'm attempting to use the stencil buffer as a clipper for my UI system, but I'm having trouble debugging a problem I'm running in to.

This is what I'm doing: A widget can pass a rectangle to the the stencil clipper functions, which will increment the stencil buffer values that it covers. Then it will draw its children, which will only get drawn in the stencilled area (so that if they extend outside they'll be clipped). After a widget is done drawing its children, it pops that rectangle from the stack and in the process decrements the values in the stencil buffer that it has previously incremented.

The slightly simplified code is below:

static void drawStencil(Rect& rect, unsigned int ref)
{
    // Save previous values of the color and depth masks
    GLboolean colorMask[4];
    GLboolean depthMask;
    glGetBooleanv(GL_COLOR_WRITEMASK, colorMask);
    glGetBooleanv(GL_DEPTH_WRITEMASK, &depthMask);

    // Turn off drawing
    glColorMask(0, 0, 0, 0);
    glDepthMask(0);

    // Draw vertices here
    ...

    // Turn everything back on
    glColorMask(colorMask[0], colorMask[1], colorMask[2], colorMask[3]);
    glDepthMask(depthMask);

    // Only render pixels in areas where the stencil buffer value == ref
    glStencilFunc(GL_EQUAL, ref, 0xFF);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
}

void pushScissor(Rect rect)
{
    // increment things only at the current stencil stack level
    glStencilFunc(GL_EQUAL, s_scissorStack.size(), 0xFF);
    glStencilOp(GL_KEEP, GL_INCR, GL_INCR);

    s_scissorStack.push_back(rect);
    drawStencil(rect, states, s_ScissorStack.size());
}

void popScissor()
{
    // undo what was done in the previous push, 
    // decrement things only at the current stencil stack level
    glStencilFunc(GL_EQUAL, s_scissorStack.size(), 0xFF);
    glStencilOp(GL_KEEP, GL_DECR, GL_DECR);

    Rect rect = s_scissorStack.back();
    s_scissorStack.pop_back();
    drawStencil(rect, states, s_scissorStack.size());
}

And this is how it's being used by the Widgets

if (m_clip)
    pushScissor(m_rect);

drawInternal(target, states);
for (auto child : m_children)
    target.draw(*child, states);

if (m_clip)
    popScissor();

This is the result of the above code:

This is the result of the above code

There are two things on the screen, a giant test button, and a window with some buttons and text areas on it. The text area scroll box is set to clip its children (so that the text doesn't extend outside the scroll box). The button is drawn after the window and should be on top of it completely. However, for some reason the text area is appearing on top of the button. The only reason I can think of that this would happen is if the stencil values were not getting decremented in the pop, and when it comes time to render the button, since those pixels don't have the right stencil value it doesn't draw over. But I can't figure out whats wrong with my code that would cause that to happen.

© Game Development or respective owner

Related posts about opengl

Related posts about stencil