'Best way for logging CPU & GPU utilization every second in linux

I want to get the CPU and GPU utilisation of my cuda program and plot them like this. What's the best way?

Here is my script:

### [1] Running my cuda program in background
./my_cuda_program &
PID_MY_CUDA_PROGRAM=$!

### [2] Getting CPU & GPU utilization in background
sar 1 | sed --unbuffered -e 's/^/SYSSTAT:/' &
PID_SYSSTAT=$!
nvidia-smi --format=csv --query-gpu=timestamp,utilization.gpu -l 1 \
    | sed --unbuffered -e 's/^/NVIDIA_SMI:/' &
PID_NVIDIA_SMI=$!

### [3] waiting for the [1] process to finish,
###     and then kill [2] processes
wait ${PID_MY_CUDA_PROGRAM}
kill ${PID_SYSSTAT}
kill ${PID_NVIDIA_SMI}
exit

That output:

SYSSTAT:Linux 4.15.0-176-generic (ubuntu00)   05/06/22        _x86_64_        (4 CPU)
NVIDIA_SMI:timestamp, utilization.gpu [%]
NVIDIA_SMI:2022/05/06 23:57:00.245, 7 %
SYSSTAT:
SYSSTAT:23:57:00        CPU     %user     %nice   %system   %iowait    %steal     %idle
SYSSTAT:23:57:01        all      8.73      0.00      5.74      7.48      0.00     78.05
NVIDIA_SMI:2022/05/06 23:57:01.246, 1 %
SYSSTAT:23:57:02        all     23.31      0.00      6.02      0.00      0.00     70.68
NVIDIA_SMI:2022/05/06 23:57:02.246, 16 %
SYSSTAT:23:57:03        all     25.56      0.00      3.76      0.00      0.00     70.68
NVIDIA_SMI:2022/05/06 23:57:03.246, 15 %
SYSSTAT:23:57:04        all     22.69      0.00      6.48      0.00      0.00     70.82
NVIDIA_SMI:2022/05/06 23:57:04.246, 21 %
SYSSTAT:23:57:05        all     25.81      0.00      3.26      0.00      0.00     70.93

it's a bit annoying to parse the log above.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source