Getting the derivative of two functions g(f(x)) and h(f(x)) in one backward pass usin Pytorch

Question

So I am passing an input x through a pretty large model f to get f(x). Then I use the output and pass it through one model g (including a loss function) to get g(f(x)) and another model h(f(x)). Now I want to compute the derivatives of g(f(x)) wrt to x and h(f(x)) wrt to x. But I need both gradients separately, so I can't just add them and do it in one backward pass.

Currently I am using:

grad_f = torch.autograd.grad(loss_g, x, retain_graph=True)[0]
grad_g = torch.autograd.grad(loss_h, x)[0]

But this is obviously quite slow, so I was wondering if there is a more efficient way to get both gradients in one combined backward pass?