Processing incoming websocket stream in python

Question

There is a server written in Go that sends me a video stream via WebSocket. I want to capture this stream in a Python program and use OpenCV to process it. However, the incoming stream seems to have a specific type of codec that makes it unable for me to process it.

Here is my code snippet that connects to the WebSocket stream:

def on_open(ws):
    print("opened")

def on_message(ws, message):
    print(message)

def on_close(ws):
    print("closed connection")

if __name__ == '__main__':
    socket = "ws://192.168.1.104:8080/stream/demo2/channel/0/mse?uuid=demo2&channel=0"
    ws = websocket.WebSocketApp(socket,
                                on_open=on_open,
                                on_message=on_message,
                                on_close=on_close)
    ws.run_forever()

And here is some part a single message printed out:

b'\x00\x00\x00\xa8moof\x00\x00\x00\x10mfhd\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x90traf\x00\x00\x00\x14tfhd\x00\x02\x00 \x00\x00\x00\x01\x01\x01\x00\x00\x00\x00\x00\x14tfdt\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\xc6\xfc\x00\x00\x00`trun\x00\x00\x0b\x05\x00\x00\x00\x06\x00\x00\x00\xb0\x02\x00\x00\x00\x00\x00\x0b\xf4\x00\x00\r8\x00\x00\x00Z\x00\x00\x0b\x9a\x00\x00\ra\x00\x00\x00Z\x00\x00\r\xb6\x00\x00\r:\x00\x00\x00Z\x00

Any idea on how to capture this video and process it using OpenCV?

P.S. I have tried cv2.imdecode(message) but I got the bad argument error message.

Answer 1

Assuming that the server sends a raw binary image with a given shape and assuming it is contained in a complete frame as a blob (e.g. without frame header information), and furthermore assuming that you know the shape of the image, then you could read it like

import numpy as np

shape = (480, 640, 3)
image = np.frombuffer(message, dtype=np.uint8).reshape(shape)

# eventually, you need cv2.imdecode(), if it's jpeg etc.

It would be much better if the server would encode the image as jpeg and then in base64 and put it in a json object before sending. Then you can decode it like

import base64
import numpy as np
import cv2

image = cv2.imdecode(np.frombuffer(base64.b64decode(base64_image), dtype=np.uint8), cv2.IMREAD_UNCHANGED)

(base64_image being the string received within the json object containing the base64 encoded jpeg image)

This way, the receiver does not have to have knowledge about the image shape.