'How to enable CUDA MPS with Multiple GPUs?

I am following https://docs.nvidia.com/deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf I am trying to learn MPS

I have 4 Tesla V100 GPU.

nvidia-smi -L
GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-e439908c-81a2-a394-b25f-cd421ffa07d3)
GPU 1: Tesla V100-SXM2-32GB (UUID: GPU-986144da-b09d-581c-767a-cc432f992f02)
GPU 2: Tesla V100-SXM2-32GB (UUID: GPU-6d759b1f-564f-c335-459d-b85fc4eba6d9)
GPU 3: Tesla V100-SXM2-32GB (UUID: GPU-f558ad53-2e37-30fc-1441-1668ac76a16d)

I am trying to enable MPS as

sudo CUDA_VISIBLE_DEVICES="GPU-e439908c-81a2-a394-b25f-cd421ffa07d3,GPU-986144da-b09d-581c-767a-cc432f992f02,GPU-6d759b1f-564f-c335-459d-b85fc4eba6d9,GPU-f558ad53-2e37-30fc-1441-1668ac76a16d" nvidia-cuda-mps-control -d


ps -ef | grep mps
root     27319 17555  0 09:00 pts/1    00:00:00 grep --color=auto mps
root     35425     1  0 Feb24 ?        00:00:01 nvidia-cuda-mps-control -d
1002     54742 35425  0 Feb24 ?        00:01:08 nvidia-cuda-mps-server


fuser -v /dev/nvidia*
                     USER        PID ACCESS COMMAND
/dev/nvidia0:        root       9232 F.... nvidia-device-p
                     root      33821 F.... kubelet
                     (unknown)  54742 F...m nvidia-cuda-mps
                  
/dev/nvidia1:        root       9232 F.... nvidia-device-p
                     root      33821 F.... kubelet
                     (unknown)  54742 F...m nvidia-cuda-mps
                     
/dev/nvidia2:        root       9232 F.... nvidia-device-p
                     root      33821 F.... kubelet
                     (unknown)  54742 F...m nvidia-cuda-mps
                     
/dev/nvidia3:        root       9232 F.... nvidia-device-p
                     root      33821 F.... kubelet
                     (unknown)  54742 F...m nvidia-cuda-mps
                     
/dev/nvidiactl:      root       9232 F.... nvidia-device-p
                     root      33821 F.... kubelet
                     (unknown)  54742 F...m nvidia-cuda-mps
                     
/dev/nvidia-uvm:     (unknown)  54742 F...m nvidia-cuda-mps

When I ran multiple TF jobs, I am confused here that nvidia-cuda-mps, I can see only on device:0 only.

How can I enable MPS for all GPUs?

Thanks

kubernetesgpunvidianvidia-dockernvidia-smi


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions