'covert a webstream to bufferarry for decodeAudioData

I am working on a project were I need to override some features of the audio stream, I am using peerjs server and am using soundtouchjs for the audio modification. I have a video tag where I stream the content to and it is working fine when it comes to directly streaming the data but when I try to first decode and change the pitch value it complains that I am not sending a buffer array.

here is the call method:

    userMedia
      .getUserMedia({ video: true, audio: true })
      .then((mediaStream) => {
        currentUserVideoRef.current.srcObject = mediaStream;
        currentUserVideoRef.current.play();

        const call = peerInstance.current.call(remotePeerId, mediaStream);

        call.on("stream", (remoteStream) => {
          RemoteStreamManipulation(remoteStream);
          remoteVideoRef.current.srcObject = remoteStream;
          remoteVideoRef.current.play();
        });
      });

I want to get the streamed data and decode it and change the pitch and some other characterestics and then send it.


    audioCtx.decodeAudioData(
      audioCtx.createMediaElementSource(currentUserVideoRef),
      (audioBuffer) => {
        shifter = new PitchShifter(audioCtx, audioBuffer, 1024);
        shifter.on("play", (detail) => {
          // do something with detail.timePlayed;
          // do something with detail.formattedTimePlayed;
          // do something with detail.percentagePlayed
        });
        shifter.tempo = 1;
        shifter.pitch = 1;
      }
    );

I also tried var data = window.URL.createObjectURL(mediaStream); to retrieve the data and make a buffer array from it but it says that the data give is not valid input for the function.

further more the buffer value is null when it comes to the remoteStream audio (doing the entire process on the other client) like below:

      call.on("stream", (remoteStream) => {
          var audioCtx = new AudioContext();
          var audioSource = audioCtx.createBufferSource(remoteStream);
          console.log(audioSource);
          console.log(audioSource.buffer);
          ...


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source