'RDMA: Can I safely remove spinlocks from mlx4_post_send() function for one sided RDMA operations?
I am porting linux-rdma to a platform with minimal OS support, e.g., spinlocks are not supported. My application aims to have application-threads and a worker-thread. The application-threads create and enqueue requests to a shared queue. The worker-thread continuously polls the queue and posts the requests.
Since I am accessing RDMA interface from a single-thread (the worker-thread), I am wondering if removing the spinlocks inside the mlx4_post_send(), mlx4_post_recv() and mlx4_poll_cq() is safe?
In case I would like to avoid the use of pthread_spinlocks_t while having many connections (ideally one QP per thread), what would be the best workaround? My application does not involve data races -- namely, each thread will only invoke send()/post() for its own QP.
Thanks, -dimitra
Solution 1:[1]
Yes, it is safe. If you check the code for the mlx5 provider you can see that single-threaded mode was added (MLX5_SINGLE_THREADED=1). I guess nobody went back and added it to the older mlx4 provider.
The official way to handle this would be to use ibv_alloc_td and then allocate the QPs in that thread domain, but that doesn't really help in your case.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Greg Inozemtsev |