How to calculate the distance vector from texel to the closest triangle edge to be saved in a FBO texture

Question

I have a triangle which I pass through a vertex and fragment shader. I want to draw per texel a correction vector u (encoded in r and b channel) that points onto the closest edge (or even one pixel further) in the coordinate space of the vertices (not texture coordinates): u is the distance between the texel (center?) and the edge.

This texture is later used as a look up table for particles to calculate a reaction. I will use u for a position correction and determining the reflection vector. This already works for hardcoded values. The texel depending u is the last piece. The shader must be OpenGL ES friendly.

For each vertex v which I pass to the vertex shader I (if it helps) could also pass the normal of each connected edge and the center of the triangle.

Any idea how this can be done?

Answer 1

A signed distance from a point P to a segment (A,B) is given by:

d = area(A, B, P) / length(B - A)

where

area(A, B, C) = (B.x - A.x)*(C.y - A.y) - (B.y - A.y)*(C.x - A.x)

Thus the distance you're looking for is given by:

dist0 = area(P, B, C) / length(C - B)
dist1 = area(A, P, C) / length(A - C)
dist2 = area(A, B, P) / length(B - A)
minimum distance = min(dist0, dist1, dist2)

The remaining question is how to map this into the GPU pipeline.

One way is to calculate vec3 dist at each vertex of a triangle, then let OpenGL interpolate it across:

dist at A = vec3( area(A, B, C)/length(C - B), 0, 0 )
dist at B = vec3( 0, area(A, B, C)/length(A - C), 0 )
dist at C = vec3( 0, 0, area(A, B, C)/length(B - A) )

Then in the fragment shader you only need to calculate the minimum of the interpolated dist components:

in vec3 dist; // interpolated from the above
minimum distance = min(dist.x, min(dist.y, dist.z))

Alternatively, if you already have the barycentric coordinates available (e.g. through the NV_fragment_shader_barycentric extension), you can piggyback on that. Notice that the barycentric coordinates formula is identical to the above, up to the scaling factors:

bary0 = area(P, B, C) / area(A, B, C)
bary1 = area(A, P, C) / area(A, B, C)
bary2 = area(A, B, P) / area(A, B, C)

Thus, if bary is already available, we only need to scale it by:

dist0 = bary0 * area(A, B, C) / length(C - B)
dist1 = bary1 * area(A, B, C) / length(A - C)
dist2 = bary2 * area(A, B, C) / length(B - A)

Since the scaling factors are constant across the triangle, this can shave off a few FLOPs.

---

EDIT: I missed that you asked for the actual vector rather than just the distance. The concept remains the same though: you'll need to pass either the normals or the vertices into the shader and based on the computed distances (using either of the above methods), select the normal of the closest edge, scale, and return it. Let's say that n0, n1, n2 are the normals opposite to vertices A, B, C correspondingly; then you'd do the following in the fragment shader:

in vec3 dist; // interpolated from the above
in vec2 n0, n1, n2; // normals
...

vec2 result;
if(dist.x < min(dist.y, dist.z))
    result = dist.x*n0;
else if(dist.y < dist.z)
    result = dist.y*n1;
else
    result = dist.z*n2;