'How to mask a shape with multiple masks

enter image description here

I want to mask a rectangle shape with two different shapes

Shape 1 Which is in shape of a circle

Shape 2 Which is in shape of rectangle

This is the code for setting stencil mask for shape 1

glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
glStencilFunc(GL_NEVER, 1, 1); // never pass stencil test
glStencilMask(1);

This is the code for setting stencil mask for shape 2

glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
glStencilFunc(GL_NEVER, 4, 4); // never pass stencil test
glStencilMask(4);

Now i can mask the geometry shape with a value of 5 and the shape will only be drawn where both the masks were drawn

 glStencilFunc(GL_EQUAL, 5, 5);
 break;

But how do i draw the geometry where Shape 1 is GL_EQUAL and Shape 2 is GL_NOTEQUAL

Like the image below

enter image description here



Solution 1:[1]

Instead of asking where "Shape 2 is GL_NOTEQUAL", ask what value it will be EQUAL there. If you're interested in pixels that are in Shape 1 and not in Shape 2, then the value of the stencil buffer there will be 1; that is 1 for Shape 1 plus 0 for Shape 2. You can do that test by modifying the ref argument to glStencilFunc:

glStencilFunc(GL_EQUAL, 1, 5);

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Yakov Galka