Creating a Google Maps-like view?

Question

I have a server feeding tiled geographic images from a GIS database, to my Android client app. I want users to be able to pan/zoom into the map like they'd do in Google Maps.

But there's an issue implementing it the standard way, with a canvas and ImageView: canvases need to have a fixed width/height, whereas Geotiff maps wraps around the globe and is thus theoretically unlimited (or at least tens of thousands of pixels wide). I.e. if our source is a Geotiff of the Northeastern USA, display a satellite view of Boston, and the user wants to pan over to New York City, then zoom into the Statue of Liberty.

Right now I have it sort of working, where:

1. when the user pans/zooms and releases their finger, I obtain the pixel coordinates of the top left and bottom right corners of the ImageView.
2. Then I tell the server to get me the tiled map at those coordinates (after converting the viewport coordinates to image coordinates of the full Geotiff image)
3. Draw the new tile while resetting the ImageView's internal transformation matrix. I internally keep track of where the image's origin (top left corner) is relative to the full Geotiff image.

But it's super hacky and causes issues when trying to render icons since we're resetting the matrix each time a new tile is drawn. Plus this way it's pretty much impossible to dynamically draw tiles while the user is continuing to zoom/pan.

Something like an infinite-canvas would be best where I'm not limited by the size of a drawable. Is there a non-hacky way of achieving this using the standard set of Android containers? Or is it inevitable I do something custom with OpenGL?