Aggregating matched points for bundle adjustment using networkx union find

Question

I am trying to solve a bundle adjustment problem. I have calculated an array that contains point correspondences for every image that overlaps.

matchMat = getMatchMat(imgs,occupancy)#matchMat[i,j]=flann.knnMatch(imgI,imgj)

I have been attempting to replicate the results of this example which has already estimated camera parameters, 3D points and has a unique index for points across all camera views.: https://scipy-cookbook.readthedocs.io/items/bundle_adjustment.html

I am starting from scratch with only images. It seems that a necessary step to bundle adjustment is being able to find which images a point is visible in. I have done some reading and one reference mentioned using union-find. If I understand correctly the problem formulation is that matching key points in each image form disjoint sets, and union-find can be used to aggregate the sets?

Here is a toy example:

kpI1 = ["a","b","c"]#key points in image 1
kpI2 = ["d","e","f"]#key points in image 2
kpI3 = ["g","h","i","j"]#key points in image 3

#hypothetical matches after running opencv matcher on pairwise on 3 overlapping images
matchMat = [ 
              [ [("a","a"),("b","b"),("c","c")]   [("a","d"),("b","e")]            [("a","g"),("b","h")] ]
              [ [("d","a"),("e","b") ]            [("d","d"),("e","e"),("f","f")]  [("d","g"),("f","i")] ]
              [ [("g","a"),("h","b") ]            [("g","d"),("i","f")]            [("g","g"),("h","h"),etc] ]
           ]

#graph representation
import networkx as nx
G.add_nodes_from(["a","b","c","d","e","f","g","h","i","j"])
G.add_edges_from([("a","d"),("a","g"),("b","h"),("b","e"),("g","d"),("f","i")])
import matplotlib.pyplot as plt
nx.draw(G, with_labels=True, font_weight='bold')
plt.show()

From this visualization, I can clearly see which points are associated but I'm stuck as far as actually leveraging this graph.

Im hoping to end up with something like this where each point can be associated with a camera

                #imageIDX   #pointIDX    #X_estimatedInPanoFrame    #y_estimatedInPanoFrame

resultArr = [[  0,           0           x1                          y1], 
             [  1,           0           x2                          y2],
             [  2,           0           x3                          y3],
             [  1,           1           x4                          y4],
             [  2,           1           x5                          y5],
             [  0,           2           x6                          y6],
             [  1,           2           x7                          y7],
             [  2,           2           x8                          y8],
             [  2,           3           x9                          y9],
             [  0,           4           x10                         y10]]

assuming dga is aggregated [by union-find?] to point 0; f,i are aggregated to 1; hbe aggregated into point 2; j 3; c 4; etc.

I just cant seem to figure out how to actually make things aggregate the way I want. What I've tried:

u = UnionFind(G.edges)
for s in u.to_sets():
    print(s)
    
{('a', 'd')}
{('a', 'g')}
{('b', 'h')}
{('b', 'e')}
{('d', 'g')}
{('f', 'i')}

#try and merge all sets that involve "a" with "a" etc
for n in G.nodes:
    u.union(n)

for s in u.to_sets():
    print(s)
    
{('a', 'd')}
{('a', 'g')}
{('b', 'h')}
{('b', 'e')}
{('d', 'g')}
{('f', 'i')}
{'a'}
{'b'}
{'c'}
{'d'}
{'e'}
{'f'}
{'g'}
{'h'}
{'i'}
{'j'}