Generative Model, but sampling from quantities with physical meaning?

Question

To my understanding the canonical generative models (GAN, VAE) sample data from what is basically random noise, or more specifically for the latter from a learned latent distribution which we have no real control of. The generative process is usually to take a value randomly, put it into the model, and get data back.

But I want this random value to have a meaning and to be constrained by my needs.

I work in particle physics, but for analogy let's say that my data is a collection of photos of bullet impact on bulletproof glass (reality: particle interaction in a detector)

The glass shattering will of course depend from the bullet type, but also from the bullet velocity and angle of incidence. One categorical variable and two continuous ones.

I want to generate more photos, but in the new photo I want to control the angle of the bullet and its velocity. In other words, I want to tell the model: "give me impacts of a .22 cal striking at 900 m/s and 10° incidence" and get in return photos that could have resulted from such an impact.

How do I do that?

I realise there may be established techniques but I am lacking the keywords or names to search for them. Any starter is much welcome