Color a LiDAR pointcloud with a segmented image

Question

I am working in a simulation environment and I have both a LiDAR and a camera placed on a certain vehicle.

The LiDAR is giving me a pointcloud in .txt format with xyz and intensity (XYZI). The camera is giving me an image (.png) and I am feeding that image into a semantic segmentation pipeline able to segment each camera image.

I would like to associate each pixel of the segmented image to a point (or a set of points) of the LiDAR Pointcloud. In brief, I would like to get a pointcloud that is colored with the same colors of the segmented image. Remember that I know exactly where each sensor is with respect to each other and the intrinsic parameters of the sensors.

The problem should have a solution in my opinion but since I was just getting started in looking into it I would like to know if you have any pointers, links to theory, suggestions, GitHub repositories that might be useful.

I am using the pointcloud library in C++.

Thanks for your help!