iPad Pro Lidar - Export Geometry & Texture

Question

I would like to be able to export a mesh and texture from the iPad Pro Lidar.

There's examples here of how to export a mesh, but Id like to be able to export the environment texture too

ARKit 3.5 – How to export OBJ from new iPad Pro with LiDAR?

ARMeshGeometry stores the vertices for the mesh, would it be the case that one would have to 'record' the textures as one scans the environment, and manually apply them?

This post seems to show a way to get texture co-ordinates, but I can't see a way to do that with the ARMeshGeometry: Save ARFaceGeometry to OBJ file

Any point in the right direction, or things to look at greatly appreciated!

Chris

Answer 1

You need to compute the texture coordinates for each vertex, apply them to the mesh and supply a texture as a material to the mesh.

let geom = meshAnchor.geometry
let vertices = geom.vertices 
let size = arFrame.camera.imageResolution
let camera = arFrame.camera
    
let modelMatrix = meshAnchor.transform
 
let textureCoordinates = vertices.map { vertex -> vector_float2 in
    let vertex4 = vector_float4(vertex.x, vertex.y, vertex.z, 1)
    let world_vertex4 = simd_mul(modelMatrix!, vertex4)
    let world_vector3 = simd_float3(x: world_vertex4.x, y: world_vertex4.y, z: world_vertex4.z)
    let pt = camera.projectPoint(world_vector3,
        orientation: .portrait,
        viewportSize: CGSize(
            width: CGFloat(size.height),
            height: CGFloat(size.width)))
    let v = 1.0 - Float(pt.x) / Float(size.height)
    let u = Float(pt.y) / Float(size.width)
    return vector_float2(u, v)
}

// construct your vertices, normals and faces from the source geometry 
// directly and supply the computed texture coords to create new geometry
// and then apply the texture.
let scnGeometry = SCNGeometry(sources: [verticesSource, textureCoordinates, normalsSource], elements: [facesSource])

let texture = UIImage(pixelBuffer: frame.capturedImage)
let imageMaterial = SCNMaterial()
imageMaterial.isDoubleSided = false
imageMaterial.diffuse.contents = texture
scnGeometry.materials = [imageMaterial]
let pcNode = SCNNode(geometry: scnGeometry)

pcNode if added to your scene will contain the mesh with the texture applied.

Texture coordinates computation from here

Answer 2

Check out my answer over here

It's a description of this project: MetalWorldTextureScan which demonstrates how to scan your environment and create a textured mesh using ARKit and Metal.