Rectangles cloud registration (shapes matching, aligning), Python prefered

Question

first, I apologize for any misuse of vocabulary, I have a hard time finding the correct keywords for my problem.

I want to compare lists of 2D coordinates that represents surfaces of rectangles, from 0 to ~20 (not thousands). The rectangles are supposed to match between each surfaces, but cannot be perfectly aligned: Width and height of homologous rectangles can sightly differ. Relative position of each rectangles can sightly differ. There may be a few rectangles without correspondence. I provide a graphical examples. Blue and red rectangles can be easily matched by the eye, even if they are not identical.

I want to identify the counterpart rectangles. I assume it should be done by finding the alignment that minimize overall transformations: unmatched rectangles, translation and scaling and then deciding of a distance threshold to accept or reject homology (like, deciding if the smalls red and blue non overlapping rectangles in bottom-left are homologous or not).

I could not identify an already implemented solution of my problem. I tried a bit with some ICP methods using the rectangle corners as landmarks, but it does not work well as it will not enforce the integrity and shape of each rectangles. I guess that by adding points on the edges of each rectangles, it should mitigate this possibility, but it seems a bit crude and inefficient approach.

Tackling the issue myself will take me time and is likely going to be sub-optimal. Is anyone aware of already implemented, robust solutions (preferentially in Python) ?