IB: Add vmw_pvrdma driver
authorAdit Ranadive <aditr@vmware.com>
Mon, 3 Oct 2016 02:10:22 +0000 (19:10 -0700)
committerDoug Ledford <dledford@redhat.com>
Wed, 14 Dec 2016 19:55:10 +0000 (14:55 -0500)
commit29c8d9eba550c6d73d17cc1618a9f5f2a7345aa1
treee8cef3b3035f1f3c4ad12f0a25ec57b6c789bcab
parentb1226c7db1d997fa6955cd3b54ba333bd0d8a29c
IB: Add vmw_pvrdma driver

This patch series adds a driver for a paravirtual RDMA device. The
device is developed for VMware's Virtual Machines and allows existing RDMA
applications to continue to use existing Verbs API when deployed in VMs
on ESXi. We recently did a presentation in the OFA Workshop [1] regarding
this device.

Description and RDMA Support
============================
The virtual device is exposed as a dual function PCIe device. One part
is a virtual network device (VMXNet3) which provides networking properties
like MAC, IP addresses to the RDMA part of the device. The networking
properties are used to register GIDs required by RDMA applications to
communicate.

These patches add support and the all required infrastructure for
letting applications use such a device. We support the mandatory Verbs API as
well as the base memory management extensions (Local Inv, Send with Inv and
Fast Register Work Requests). We currently support both Reliable Connected
and Unreliable Datagram QPs but do not support Shared Receive Queues
(SRQs).

Also, we support the following types of Work Requests:
 o Send/Receive (with or without Immediate Data)
 o RDMA Write (with or without Immediate Data)
 o RDMA Read
 o Local Invalidate
 o Send with Invalidate
 o Fast Register Work Requests

This version only adds support for version 1 of RoCE. We will add RoCEv2
support in a future patch. We do support registration of both MAC-based
and IP-based GIDs. I have also created a git tree for our user-level driver
[2].

Testing
=======
We have tested this internally for various types of Guest OS - Red Hat,
Centos, Ubuntu 12.04/14.04/16.04, Oracle Enterprise Linux, SLES 12
using backported versions of this driver. The tests included several
runs of the performance tests (included with OFED), Intel MPI PingPong
benchmark on OpenMPI, krping for FRWRs. Mellanox has been kind enough
to test the backported version of the driver internally on their hardware
using a VMware provided ESX build. I have also applied and tested this
with Doug's k.o/for-4.9 branch (commit 5603910b). Note, that this patch
series should be applied all together. I split out the commits so that
it may be easier to review.

PVRDMA Resources
================
[1] OFA Workshop Presentation -
https://openfabrics.org/images/eventpresos/2016presentations/102parardma.pdf

[2] Libpvrdma User-level library -
http://git.openfabrics.org/?p=~aditr/libpvrdma.git;a=summary

Reviewed-by: Jorgen Hansen <jhansen@vmware.com>
Reviewed-by: George Zhang <georgezhang@vmware.com>
Reviewed-by: Aditya Sarwade <asarwade@vmware.com>
Reviewed-by: Bryan Tan <bryantan@vmware.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Signed-off-by: Adit Ranadive <aditr@vmware.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>
19 files changed:
MAINTAINERS
drivers/infiniband/Kconfig
drivers/infiniband/hw/Makefile
drivers/infiniband/hw/vmw_pvrdma/Kconfig [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/Makefile [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma.h [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cmd.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_doorbell.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_misc.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_mr.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_ring.h [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c [new file with mode: 0644]
drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.h [new file with mode: 0644]
include/uapi/rdma/Kbuild
include/uapi/rdma/vmw_pvrdma-abi.h [new file with mode: 0644]