'OpenGL ES: Well defined way to discard primitive in vertex shader
My vertex shader decides to discard some triangles. If a triangle should be discarded, what I currently do is set the output position to NaN:
gl_Position = vec4(intBitsToFloat(int(0xFFC00000u)), intBitsToFloat(int(0xFFC00000u)), intBitsToFloat(int(0xFFC00000u)), intBitsToFloat(int(0xFFC00000u)));
intBitsToFloat(int(0xFFC00000u)) is floating point NaN.
This works on the platforms I tested it on.
However, Section 2.3.4.1 of the OpenGL ES 3.2 Spec states that
The special values Inf and −Inf encode values with magnitudes too large to be represented; the special value NaN encodes “Not A Number” values resulting from undefined arithmetic operations such as 0/0. Implementations are permitted, but not required, to support Inf's and NaN's in their floating-point computations.
So my approach might actually yield undefined behavior. What's the proper, well defined OpenGL ES 3.x way to completely discard a triangle in the vertex shader?
Solution 1:[1]
The result is required to be consistent, even if NaN is not generated. If you set all 3 vertices to NaN (or whatever the fallback value turns out to be) then they are going to be coincident locations. Any two coincident vertices in a triangle = zero area = culled. So I think this works in either case ...
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | solidpixel |
