Audio/Video syncing problem while recording and playing webm chunks on Chrome of Windows 10

Question

Problem

I am recording the webm chunks by MediaRecorder at Chrome 83 in Windows 10 and sending these to other computer. These chunks are playing on another Chrome by using Media Source Extension(MSE).

sourceBuffer.appendBuffer(webmChunkData);

Everything works fine between 1 to 1.20 seconds. But after that, the audio/video syncing problem starts. The gap between audio and video is minimal at the moment, but as time increases, the gap also rises.

The weird thing is that we can see the different behaviour on different browsers, let me show this by

Chrome's version is 83+ in almost all operating systems.

Camera can be the problem ?

I think Camera is not the problem as I have dual operating systems Fedora and Windows in the same machine. And webm chunks play fine with the Fedora.

Sample rate can be the problem ?

I doubt this. But when I compare the sample rate used by browsers while playing. chrome://media-internals shows 48000 for both with and without a syncing problem.

Warning message from Chrome

Chrome which has sync problem also shows the below message on chrome://media-internals

Question:

Why there is an audio/video syncing problem when both recording and playing are performed on Chrome browser in Windows 10?

How can I eliminate this syncing problem?

Answer 1

I believe I have a workaround for you. The problem seems specific to Chrome + MediaRecorder + VP8, and has nothing to do with MSE or the platform. I have the same issues on Chrome 98 on Mac 12.2.1. Additionally, if you decrease the .start(timeslice) argument, the issue will appear more rapidly and more severely.

However... when I use VP9 the problem does not manifest!

I'm using code like this:

function supportedMimeTypes(): string[] {
  // From least desirable to most desirable
  return [
    // most compatible, but chrome creates bad chunks
    'video/webm;codecs=vp8,opus',
    // works in chrome, firefox falls back to vp8
    'video/webm;codecs=vp9,opus'
  ].filter(
    (m) => MediaRecorder.isTypeSupported(m)
  );
}

const mimeType = supportedMimeTypes().pop();

if (!mimeType) throw new Error("Could not find a supported mime-type");

const recorder = new MediaRecorder(stream, {
  // be sure to use a mimeType with a specific `codecs=` as otherwise
  // chrome completely ignores it and uses video/x-matroska!
  // https://stackoverflow.com/questions/64233494/mediarecorder-does-not-produce-a-valid-webm-file
  mimeType,
});

recorder.start(1000)

The resulting VP9 appears to play in Firefox, and a VP8 recorded in Firefox plays well in Chrome too.