Keras: Loss for image rotation and translate (target errore registration)?

Question

My model return 3 cordinate [x,y,angle]. I want TRE similarity between 2 images. My custom loss is:

loss(y_true, y_pred):

  s = tfa.image.rotate(images=y_true[0], angles=y_pred[0][0])
  s = tfa.image.translate(images=s, translations=y_pred[0][1:])
  s = tf.reduce_sum(tf.sqrt(tf.square(s-y_true[1])))

y_pred=(1, 3)->tensor with [angle,x,y]

y_true=(2,128,128)-> in y_true[0] and y_true[1]: image. I:

• s=Rotate and translate y_true[0],
• Compare s and y_true[1], with MSE

I can't use tfa.image.translate beacuse is not differentiable? How can rotate an image in a custom loss function? There are problem with gradient?

Answer 1

I Believe this will or will not work depending on the frequency distribution in your data. But in fft space this might be easier.