Find matching point epipolar geometry

Question

I'm trying to determine the position of an object, seen on two images at the same time, in python. I already found the fundamental matrix of the situation thanks to openc and now, what I'm trying to do, is chosing a point in an image and finding the matching point on the second image with the help of the fundamental matrix, but I don't find anything on internet, or in the opencv library.

Does anyone have a solution ?

Answer 1

Based on the tutorial here, first, we need a bunch of matching points to extract the fundamental matrix. So, to show how it works, I created a synthetic set of matching points to extract the fundamental matrix. Then, I show you how one can use the fundamental matrix to extract the epilines in the second image, for any given point in the first image. If the point from the first image represents an object, the object is located somewhere on the epiline in the second image.

So, I start with two synthetic empty images in RGBA, a set of arbitrary points in the first image, and an arbitrary transformation function transform_fn to map points on the first image onto the second image:

import numpy as np
import cv2 as cv

img1 = np.ones((64, 64, 4), dtype=np.uint8) * 255
img2 = np.ones_like(img1, dtype=np.uint8) * 255

pts1 = [
    [10, 20],
    [25, 20],
    [28, 18],
    [30, 15],
    [50, 40],
    [53, 43],
    [55, 45],
    [18, 58],
]
transform_fn = lambda pt: [
    int(pt[0]*.5)+int(pt[1]*.2) + 5, 
    int(pt[0]*.2)-int(pt[1]*.4) + 25,
]

pts2 = [transform_fn(pt) for pt in pts1]

# paint black
for (i, j) in pts1:
    img1[i, j, :-1] = 0 

# paint black
for (i, j) in pts2:
    img2[i, j, :-1] = 0 
    
pts1 = np.int32(pts1)
pts2 = np.int32(pts2)

With these paired points (pts1 and pts2), we can the extract fundamental matrix:

F, _ = cv.findFundamentalMat(pts1, pts2, cv.FM_LMEDS)

F above is the fundamental matrix. Before showing how F works, I need an arbitrary point in the first image that is synthetically mapped to a point in the second image as the gold standard for evaluation and demo.

I chose pt1 = [32, 32] to demonstrate this. Using transform_fn, the location of this point in second image must be pt2 = transform_fn(pt1). I paint them red, and plot the images:

pt1 = [32, 32]
pt2 = transform_fn(pt1)

# paint red
img1[pt1[0], pt1[1], :] = np.array([255, 0, 0, 255])
img2[pt2[0], pt2[1], :] = np.array([255, 0, 0, 255])


from matplotlib import pyplot as plt

def plot(img1, img2): 
    fig, axs = plt.subplots(1, 2, )

    axs[0].imshow(img1)
    axs[0].set_xticks([])
    axs[0].set_yticks([])
    axs[0].set_xlabel('img1')

    axs[1].imshow(img2)
    axs[1].set_xticks([])
    axs[1].set_yticks([])
    axs[1].set_xlabel('img2')

    plt.show()

plot(img1, img2)

Now, all we need to do is to use F to find the epiline corresponding to point pt1. We expect pt2 to be on the epiline. Here is the demostration:

lines2 = cv.computeCorrespondEpilines(np.array([pt1]), 1, F)
lines2 = lines2.reshape(-1,3)
lines2

_, c, _ = img1.shape
for r in lines2:
    j0, i0 = map(int, [0, -r[2]/r[1] ])
    j1, i1 = map(int, [c, -(r[2]+r[0]*c)/r[1] ])
    
    img2 = cv.line(img2, (i0, j0), (i1, j1), [255, 255, 0, 127], 1) # yello line
    img2[pt2[0], pt2[1], :] = np.array([255, 0, 0, 255]) # red-dot (true location)

plot(img1, img2)