OffsetImage zoom factor relative to axes coordinates

Question

I am trying to add an image with a given pixel size to a plot within an AnnotationBbox.

Here is an example of what I am trying to do with an image of size 200px x 200px.

import numpy as np
import matplotlib.pylab as plt
from matplotlib.offsetbox import AnnotationBbox
from matplotlib.offsetbox import OffsetImage

img = np.zeros((200, 200, 4))
img[..., 3] = 1.

fig = plt.figure(dpi=150)
ax = fig.add_subplot(111, aspect=1.)

im = OffsetImage(img, zoom=1)
ax.add_artist(AnnotationBbox(im, (0, 0), pad=0, frameon=False))

ax.set_xlim(-2., 2.)
ax.set_ylim(-2., 2.)

fig.tight_layout()

This produces the following output.

I need the image to be of a given size in axes coordinates. Let's say 2x2.

Right now I achieve this by changing the zoom factor of the OffsetImage via trial and error. However, the image stays fixed in size relative to the figure, not to the axes coordinates, which can be seen by changing the limits.

fig = plt.figure(dpi=150)
ax = fig.add_subplot(111, aspect=1.)
im = OffsetImage(img, zoom=1)
ax.add_artist(AnnotationBbox(im, (0, 0), pad=0, frameon=False))
ax.set_xlim(-4., 4.)
ax.set_ylim(-4., 4.)
fig.tight_layout()

I fail to figure out the relation between zoom factor, pixel size of the image, size of the figure, and potentially the dpi, to have an image that has a fixed size in axes coordinates independent of the limits.

How can I achieve this?