JComponent JPanel Zoom, pan and Coordinates Question

I have a JPanel with an vector image that the user can zoom and pan on. Overlaying this image is a (transp开发者_如何学Goarent) JComponent, which I allow the user to annotate the underlying image. This works great at full scale, but If I zoom in, using AffineTransform, the overlaying coordinates are affected also. So, If a user draws a box on the image the box is scaled also.

Any suggestions on how to decouple this behaviour? So that the JComponent is not affected by the JPanel's AffineTransform?

---

Typically, you need both a forward and inverse transform to translate between the two co-ordinate systems. In this example, the scaling equations are explicit; in this alternate approach, a second AffineTransform is used.