drm/amdgpu: Don't pin VRAM without DMABUF_MOVE_NOTIFY

Pinning of VRAM is for peer devices that don't support dynamic attachment
and move notifiers. But it requires that all such peer devices are able to
access VRAM via PCIe P2P. Any device without P2P access requires migration
to GTT, which fails if the memory is already pinned for another peer
device.

Sharing between GPUs should not require pinning in VRAM. However, if
DMABUF_MOVE_NOTIFY is disabled in the kernel build, even DMABufs shared
between GPUs must be pinned, which can lead to failures and functional
regressions on systems where some peer GPUs are not P2P accessible.

Disable VRAM pinning if move notifiers are disabled in the kernel build
to fix regressions when sharing BOs between GPUs.

Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Tested-by: Hao (Claire) Zhou <hao.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index 0446586bd5a706d31c548919952c4a2176df3d0b..5740e8d1a5226e54267a247e84369713a359afd0 100644 (file)
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -81,14 +81,21 @@ static int amdgpu_dma_buf_pin(struct dma_buf_attachment *attach)
 
        dma_resv_assert_held(dmabuf->resv);
 
-       /*
-        * Try pinning into VRAM to allow P2P with RDMA NICs without ODP
+       /* Try pinning into VRAM to allow P2P with RDMA NICs without ODP
         * support if all attachments can do P2P. If any attachment can't do
         * P2P just pin into GTT instead.
+        *
+        * To avoid with conflicting pinnings between GPUs and RDMA when move
+        * notifiers are disabled, only allow pinning in VRAM when move
+        * notiers are enabled.
         */
-       list_for_each_entry(attach, &dmabuf->attachments, node)
-               if (!attach->peer2peer)
-                       domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
+       if (!IS_ENABLED(CONFIG_DMABUF_MOVE_NOTIFY)) {
+               domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
+       } else {
+               list_for_each_entry(attach, &dmabuf->attachments, node)
+                       if (!attach->peer2peer)
+                               domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
+       }
 
        if (domains & AMDGPU_GEM_DOMAIN_VRAM)
                bo->flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;