How to compare two images without comparing colors in open CV?

Question

I am trying implement a logic using Open CV which finds the difference between two images by ignoring the color differences:

The following two images are of exactly same size. In the following images I want to get the difference for only blue bar and ignore the red and green bars since they are same with just different colors.

ImageA:

ImageB: The following code will find all the differences. Any thoughts on how we can ignore the color only changes ? The solution should be generic and not specific to these images.

image = cv2.absdiff(imageA, imageB)

By compare, I am referring to finding differences between two images. Since I am using 'absdiff', I am getting all the differences. Now the problem is I want to ignore the differences which are caused by color changes alone.

For e.g.

1. Ignore the differences - If some elements in the image have different border color and everything else is same
2. Ignore the differences - If text color in the image is different but font and size is the same etc. Hope that clarifies the question Thanks in advance!