RealityKit - How to get rendered frame?

Question

So session delegate allows me to get a frame that contains camera image + some details. But how can I get something similar from RealityKit. Like a frame of rendered objects + shadows without background?

I'd like to do my own post-frame rendering in metal. So have realityKit render nice meshes and then do adjustments to the resulting frame myself & render to my own surface.

Regards Dariusz

Answer 1

TLDR

You can add a session delegate to RealityKit's ARView: arView.session.delegate = sessionDelegate

SwiftUI

The Following is a how you could implement a session delegate with SwiftUI:

class SessionDelegate<ARContainer: UIViewRepresentable>: NSObject, ARSessionDelegate {
    var arVC: ARContainer
    var arGameView : ARView?
    
    init(_ control: ARContainer) {
        self.arVC = control
    }
    
    func setARView(_ arView: ARView){
        arGameView = arView // get access the arView
    }
    
    func session(_ session: ARSession, didUpdate anchors : [ARAnchor]) {}
    func session(_ session: ARSession, didUpdate frame: ARFrame) {
       print("frame", frame)    
    }
    func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {}
    func session(_ session: ARSession, didRemove anchors: [ARAnchor]) {}
}

Make your delegate available to your UIViewRepresentable through the makeCoordinator function:

struct CameraARContainer: UIViewRepresentable {
    
    func makeCoordinator() -> SessionDelegate<Self>{
        SessionDelegate<Self>(self)
    }
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARGameView(frame: .zero)
        arView.session.delegate = context.coordinator
        context.coordinator.setARView(arView)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {}
}