AVPlayer audio out of sync after seek

Question

I use PlayerKit in one app to stream video from Cloudinary. Everything works fine until I use seek to move video forwards or backwards. I pause the video, use seek to move it to time I wish. After that the video plays fine but audio only starts after 20-30 seconds and is out of sync.

I've tried waiting for it to buffer, and different options on AVPlayerItem, Asset and Player but to no avail. One thing I did notice is that audio is not buffered while video is paused, it starts buffering audio only when I play the video.

PlayerKit is just a wrapper for AVPlayer and uses AVPlayer methods and listeners to inform me of buffering time and playing state so it's not up to the pod.