How do I generate a partial view of a mesh as a point cloud in Python?

Question

I have a dataset of meshes, which I want to use to generate partial view data as point clouds. In other words, simulating the way an RGB-D sensor would work.

My solution is very naive, so feel free to disregard it and suggest another method.

It consists of taking a rendered RGB and a rendered D image from an o3d visualization, as such:

vis.add_geometry(tr_mesh)

... # set some params to have a certain angle

vis.capture_screen_image('somepath.png')
vis.capture_depth_image('someotherpath.png')

These are saved as PNG files. Then I combine them into an o3d RGBDImage:

 # Load the RGB image
 rgb = o3d.io.read_image(rgb_path)

 # Load the depth image
 depth = o3d.io.read_image(depth_path)

 # Convert the RGB and depth images into pointcloud
 rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(color=o3d.geometry.Image(rgb), 
                                                           depth=o3d.geometry.Image(depth), 
                                                           convert_rgb_to_intensity=False)

And convert this to a PointCloud

pcd = o3d.geometry.PointCloud.create_from_rgbd_image(image=rgbd,
intrinsic=o3d.camera.PinholeCameraIntrinsic(o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault))

This has many limitations:

• the depth is quantized to 256 values. These are literally in 0...255 so they don't match the scale of the image height and width. Not to mention, it completely loses the original scale of the mesh object itself.
• the camera params (focal length etc) are not recreated identically so the point cloud is deformed.

Again, this is a very naive solution, so completely different approaches are very welcome.

This is not a duplicate of Can I generate Point Cloud from mesh? as I want partial views. So think that question, but with back-face culling.

Answer 1

Getting back with the solution.

No image capture is needed. There is a function called vis.capture_depth_point_cloud()

So the partial view can be generated by simply running

vis.add_geometry(tr_mesh)

... # set some params to have a certain angle

vis.capture_depth_point_cloud("somefilename.pcd")

This also has a parameter called convert_to_world_coordinate which is very useful.

There doesn't seem to be a way to change the resolution of the sensor. Though up-(or down-)scaling the object, capturing the point cloud, then down-(or up-)scaling the point cloud should obtain the same effect.