How to convert CMSampleBuffer to OpenCV's Mat instance in swift

Question

I am working on an OpenCV project where I am taking input from iPhone native camera as CMSampleBuffer now I wanted to create Mat instance that is required in OpenCV for further process

I have found some old post related with it but all are not working in current swift as those are pretty old.

Raw image data from camera like "645 PRO"

How to convert CMSampleBufferRef to IplImage (iOS)

Answer 1

First, convert CMSampleBuffer To UIImage.

extension CMSampleBuffer {
    func asUIImage()-> UIImage? {
        guard let imageBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(self) else {
            return nil
        }
        let ciImage = CIImage(cvPixelBuffer: imageBuffer)
        return convertToUiImage(ciImage: ciImage)
    }
    
    func convertToUiImage(ciImage: CIImage) -> UIImage? {
        let context = CIContext(options: nil)
        context.clearCaches()
        guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
            return nil
        }
        let image = UIImage(cgImage: cgImage)
        return image
    }
}

Then you can easily convert UIImage to Mat and return UIImage with/without doing something.
OpenCVWrapper.h file

#import <Foundation/Foundation.h>
#import <UIKit/UIKit.h>

NS_ASSUME_NONNULL_BEGIN

@interface OpenCVWrapper : NSObject

+ (UIImage *) someOperation : (UIImage *) uiImage;

@end

NS_ASSUME_NONNULL_END

OpenCVWrapper.mm file

#import <opencv2/opencv.hpp>
#import <opencv2/imgcodecs/ios.h>
#import <opencv2/core/types_c.h>
#import "OpenCVWrapper.h"
#import <opencv2/Mat.h>

#include <iostream>
#include <random>
#include <vector>
#include <chrono>
#include <stdint.h>

@implementation OpenCVWrapper

+ (UIImage *) someOperation : (UIImage *) uiImage {
    cv::Mat sourceImage;
    UIImageToMat(uiImage, sourceImage);
    // Do whatever you need
    return MatToUIImage(sourceImage);
}

@end

Answer 2

As far as I know openCV is written in C++ and you will have to interact with it using an Objc++ or an Objc wrapper. Hence the resources you found are solid and there are many others on working with OpenCV on iOS. A simple swifty wrapper I found online is - https://github.com/Legoless/LegoCV.

There is a full interoperability between Objc/Objc++ and swift and it is fairly easy to implement. See these two resources for more info - https://developer.apple.com/documentation/swift/imported_c_and_objective-c_apis/importing_objective-c_into_swift https://rderik.com/blog/understanding-objective-c-and-swift-interoperability/