Deploying object detection model for real-time inference

Question

I have an OD Model trained on a custom dataset. I would like to deploy the model as an API. The model will be used in real-time inference and I am planning on monetizing this API on one of the API marketplaces such as AWS, Rakuten's Rapid API, etc.

My concern is if the OD Model is provided as an API, performing predictions in real-time on a video stream (surveillance camera feed) will bring network latency that will make the app slower. Are there any other alternatives to solve the latency issues?

For instance, If I package the code and artifacts to be executed on the client's system, network latency can be eliminated but at the risk of exposing the model, code, etc. So API seems to be the ideal solution for my use case.

What would be the best approach to execute such a scenario?

Moreover, if pre-processing and post-processing are involved for the images. Are there any platforms that aid to package our application and converting it as a black box that takes image inputs and provides image outputs?

Answer 1

For AWS Marketplace, you can sell a Amazon SageMaker "model package" product, a pre-trained model for making predictions that does not require any further training by the buyer.

This should address your concerns on intellectual-property protection and somewhat address your concerns on latency.

Regarding intellectual-property protection, you as the seller package your model inside a Docker container. When it is deployed in the buyer's Amazon SageMaker service in their AWS account, they have no direct access to the container. They can only interact with your model via the SageMaker APIs. More info here: https://docs.aws.amazon.com/marketplace/latest/userguide/ml-security-and-intellectual-property.html

Regarding latency, the model package is deployed in the buyer's AWS account in the region of their choosing. Although the model package cannot be deployed onto edge devices, it brings it one step closer instead of where you as the seller hosts the API.

For more information on publishing Amazon SageMaker products on AWS Marketplace, see the seller guide: https://docs.aws.amazon.com/marketplace/latest/userguide/machine-learning-products.html