In computer graphics programming, a stencil buffer is an area of memory that can be drawn to and used in relation to other graphics buffers to achieve different effects, or to mask areas of a scene so they will not be processed or rendered. A stencil buffer is a per-pixel buffer similar to a black and white image, in which each location in the buffer can hold a value that is most often 1 byte long at most but can be as small as 1 bit. The buffer originally was designed to act as a true or false mask, much like a real-world stencil, so areas of a scene could be obscured or ignored by setting the values of different regions of the stencil buffer to a value other than zero. There are a number of uses for the buffer beyond simple masking, including drawing a shadow volume, implementing halos around objects or locating areas where objects overlap one another. In many instances, the buffer is maintained in the graphics memory on a graphics card so it can be accessed quickly during rendering.
The concept of a stencil buffer originally was a way to create a two-dimensional (2D) mask that could be applied to a three-dimensional (3D) scene as the vertices moved through the graphics pipeline. If the location of the 3D vertex, once rasterized, related to an area of the stencil buffer that was not set to zero, then that vertex could be ignored and the rest of its pipeline processing skipped. This allowed programmers a way to render only small parts of a scene or to render an image into an irregularly shaped viewport, such as a reflection in a small mirror within a larger scene, where the shape of the mirror is drawn as a stencil in the buffer, and then the reflection is drawn through that shape.
Memory concerns led to the original form of a stencil buffer being a grid of single-bit values in which each pixel in the buffer could only be true or false. As graphics cards developed, most buffers became able to hold a full byte value at each pixel location, allowing programmers to have more complex values in a location. This can allow the buffer to be used to detect areas where objects overlap, which can be done by incrementing the value of the buffer at a location if an object occupies that pixel on the screen when rasterized. After this operation, the value of each pixel location in the buffer will equal the number of objects that cross that pixel on a screen.
One of the most popular uses for a stencil buffer is creating volume shadows. This can be done by first rendering a scene with very dim lighting so everything appears to be shaded. Next, calculations are made to determine the location of all the polygon surfaces in the scene that are illuminated by a light source, and these shapes are transferred to the stencil buffer. The scene is rendered again as if every object is fully lit and is blended through the stencil shapes onto the shadowed scene, effectively creating an image that appears to contain objects with shadows.