Trying to use stencils in 2D while retaining layer depth
- by Steve
This is a screen of what's going on just so you can get a frame of reference.
http://i935.photobucket.com/albums/ad199/fobwashed/tilefloors.png
The problem I'm running into is that my game is slowing down due to the amount of texture swapping I'm doing during my draw call. Since walls, characters, floors, and all objects are on their respective sprite sheet containing those types, per tile draw, the loaded texture is swapping no less than 3 to 5+ times as it cycles and draws the sprites in order to layer properly.
Now, I've tried throwing all common objects together into their respective lists, and then using layerDepth drawing them that way which makes things a lot better, but the new problem I'm running into has to do with the way my doors/windows are drawn on walls. Namely, I was using stencils to clear out a block on the walls that are drawn in the shape of the door/window so that when the wall would draw, it would have a door/window sized hole in it.
This is the way my draw was set up for walls when I was going tile by tile rather than grouped up common objects.
first it would check to see if a door/window was on this wall. If not, it'd skip all the steps and just draw normally. Otherwise
end the current spriteBatch
Clear the buffers with a transparent color to preserve what was already drawn
start a new spritebatch with stencil settings
draw the door area
end the spriteBatch
start a new spritebatch that takes into account the previously set stencil
draw the wall which will now be drawn with a hole in it
end that spritebatch
start a new spritebatch with the normal settings to continue drawing tiles
In the tile by tile draw, clearing the depth/stencil buffers didn't matter since I wasn't using any layerDepth to organize what draws on top of what.
Now that I'm drawing from lists of common objects rather than tile by tile, it has sped up my draw call considerably but I can't seem to figure out a way to keep the stencil system to mask out the area a door or window will be drawn into a wall. The root of the problem is that when I end a spriteBatch to change the DepthStencilState, it flattens the current RenderTarget and there is no longer any depth sorting for anything drawn further down the line. This means walls always get drawn on top of everything regardless of depth or positioning in the game world and even on top of each other as the stencil has to happen once for every wall that has a door or window.
Does anyone know of a way to get around this? To boil it down, I need a way to draw having things sorted by layer depth while also being able to stencil/mask out portions of specific sprites.
