Commit | Line | Data |
---|---|---|
7bc7f618 XY |
1 | ================================== |
2 | VDUSE - "vDPA Device in Userspace" | |
3 | ================================== | |
4 | ||
5 | vDPA (virtio data path acceleration) device is a device that uses a | |
6 | datapath which complies with the virtio specifications with vendor | |
7 | specific control path. vDPA devices can be both physically located on | |
8 | the hardware or emulated by software. VDUSE is a framework that makes it | |
9 | possible to implement software-emulated vDPA devices in userspace. And | |
10 | to make the device emulation more secure, the emulated vDPA device's | |
11 | control path is handled in the kernel and only the data path is | |
12 | implemented in the userspace. | |
13 | ||
14 | Note that only virtio block device is supported by VDUSE framework now, | |
15 | which can reduce security risks when the userspace process that implements | |
16 | the data path is run by an unprivileged user. The support for other device | |
17 | types can be added after the security issue of corresponding device driver | |
18 | is clarified or fixed in the future. | |
19 | ||
20 | Create/Destroy VDUSE devices | |
09b6addf | 21 | ---------------------------- |
7bc7f618 XY |
22 | |
23 | VDUSE devices are created as follows: | |
24 | ||
25 | 1. Create a new VDUSE instance with ioctl(VDUSE_CREATE_DEV) on | |
26 | /dev/vduse/control. | |
27 | ||
28 | 2. Setup each virtqueue with ioctl(VDUSE_VQ_SETUP) on /dev/vduse/$NAME. | |
29 | ||
30 | 3. Begin processing VDUSE messages from /dev/vduse/$NAME. The first | |
31 | messages will arrive while attaching the VDUSE instance to vDPA bus. | |
32 | ||
33 | 4. Send the VDPA_CMD_DEV_NEW netlink message to attach the VDUSE | |
34 | instance to vDPA bus. | |
35 | ||
36 | VDUSE devices are destroyed as follows: | |
37 | ||
38 | 1. Send the VDPA_CMD_DEV_DEL netlink message to detach the VDUSE | |
39 | instance from vDPA bus. | |
40 | ||
41 | 2. Close the file descriptor referring to /dev/vduse/$NAME. | |
42 | ||
43 | 3. Destroy the VDUSE instance with ioctl(VDUSE_DESTROY_DEV) on | |
44 | /dev/vduse/control. | |
45 | ||
46 | The netlink messages can be sent via vdpa tool in iproute2 or use the | |
47 | below sample codes: | |
48 | ||
49 | .. code-block:: c | |
50 | ||
51 | static int netlink_add_vduse(const char *name, enum vdpa_command cmd) | |
52 | { | |
53 | struct nl_sock *nlsock; | |
54 | struct nl_msg *msg; | |
55 | int famid; | |
56 | ||
57 | nlsock = nl_socket_alloc(); | |
58 | if (!nlsock) | |
59 | return -ENOMEM; | |
60 | ||
61 | if (genl_connect(nlsock)) | |
62 | goto free_sock; | |
63 | ||
64 | famid = genl_ctrl_resolve(nlsock, VDPA_GENL_NAME); | |
65 | if (famid < 0) | |
66 | goto close_sock; | |
67 | ||
68 | msg = nlmsg_alloc(); | |
69 | if (!msg) | |
70 | goto close_sock; | |
71 | ||
72 | if (!genlmsg_put(msg, NL_AUTO_PORT, NL_AUTO_SEQ, famid, 0, 0, cmd, 0)) | |
73 | goto nla_put_failure; | |
74 | ||
75 | NLA_PUT_STRING(msg, VDPA_ATTR_DEV_NAME, name); | |
76 | if (cmd == VDPA_CMD_DEV_NEW) | |
77 | NLA_PUT_STRING(msg, VDPA_ATTR_MGMTDEV_DEV_NAME, "vduse"); | |
78 | ||
79 | if (nl_send_sync(nlsock, msg)) | |
80 | goto close_sock; | |
81 | ||
82 | nl_close(nlsock); | |
83 | nl_socket_free(nlsock); | |
84 | ||
85 | return 0; | |
86 | nla_put_failure: | |
87 | nlmsg_free(msg); | |
88 | close_sock: | |
89 | nl_close(nlsock); | |
90 | free_sock: | |
91 | nl_socket_free(nlsock); | |
92 | return -1; | |
93 | } | |
94 | ||
95 | How VDUSE works | |
96 | --------------- | |
97 | ||
98 | As mentioned above, a VDUSE device is created by ioctl(VDUSE_CREATE_DEV) on | |
99 | /dev/vduse/control. With this ioctl, userspace can specify some basic configuration | |
100 | such as device name (uniquely identify a VDUSE device), virtio features, virtio | |
101 | configuration space, the number of virtqueues and so on for this emulated device. | |
102 | Then a char device interface (/dev/vduse/$NAME) is exported to userspace for device | |
103 | emulation. Userspace can use the VDUSE_VQ_SETUP ioctl on /dev/vduse/$NAME to | |
104 | add per-virtqueue configuration such as the max size of virtqueue to the device. | |
105 | ||
106 | After the initialization, the VDUSE device can be attached to vDPA bus via | |
107 | the VDPA_CMD_DEV_NEW netlink message. Userspace needs to read()/write() on | |
108 | /dev/vduse/$NAME to receive/reply some control messages from/to VDUSE kernel | |
109 | module as follows: | |
110 | ||
111 | .. code-block:: c | |
112 | ||
113 | static int vduse_message_handler(int dev_fd) | |
114 | { | |
115 | int len; | |
116 | struct vduse_dev_request req; | |
117 | struct vduse_dev_response resp; | |
118 | ||
119 | len = read(dev_fd, &req, sizeof(req)); | |
120 | if (len != sizeof(req)) | |
121 | return -1; | |
122 | ||
123 | resp.request_id = req.request_id; | |
124 | ||
125 | switch (req.type) { | |
126 | ||
127 | /* handle different types of messages */ | |
128 | ||
129 | } | |
130 | ||
131 | len = write(dev_fd, &resp, sizeof(resp)); | |
132 | if (len != sizeof(resp)) | |
133 | return -1; | |
134 | ||
135 | return 0; | |
136 | } | |
137 | ||
138 | There are now three types of messages introduced by VDUSE framework: | |
139 | ||
140 | - VDUSE_GET_VQ_STATE: Get the state for virtqueue, userspace should return | |
141 | avail index for split virtqueue or the device/driver ring wrap counters and | |
142 | the avail and used index for packed virtqueue. | |
143 | ||
144 | - VDUSE_SET_STATUS: Set the device status, userspace should follow | |
145 | the virtio spec: https://docs.oasis-open.org/virtio/virtio/v1.1/virtio-v1.1.html | |
146 | to process this message. For example, fail to set the FEATURES_OK device | |
147 | status bit if the device can not accept the negotiated virtio features | |
148 | get from the VDUSE_DEV_GET_FEATURES ioctl. | |
149 | ||
150 | - VDUSE_UPDATE_IOTLB: Notify userspace to update the memory mapping for specified | |
151 | IOVA range, userspace should firstly remove the old mapping, then setup the new | |
152 | mapping via the VDUSE_IOTLB_GET_FD ioctl. | |
153 | ||
154 | After DRIVER_OK status bit is set via the VDUSE_SET_STATUS message, userspace is | |
155 | able to start the dataplane processing as follows: | |
156 | ||
157 | 1. Get the specified virtqueue's information with the VDUSE_VQ_GET_INFO ioctl, | |
158 | including the size, the IOVAs of descriptor table, available ring and used ring, | |
159 | the state and the ready status. | |
160 | ||
161 | 2. Pass the above IOVAs to the VDUSE_IOTLB_GET_FD ioctl so that those IOVA regions | |
162 | can be mapped into userspace. Some sample codes is shown below: | |
163 | ||
164 | .. code-block:: c | |
165 | ||
166 | static int perm_to_prot(uint8_t perm) | |
167 | { | |
168 | int prot = 0; | |
169 | ||
170 | switch (perm) { | |
171 | case VDUSE_ACCESS_WO: | |
172 | prot |= PROT_WRITE; | |
173 | break; | |
174 | case VDUSE_ACCESS_RO: | |
175 | prot |= PROT_READ; | |
176 | break; | |
177 | case VDUSE_ACCESS_RW: | |
178 | prot |= PROT_READ | PROT_WRITE; | |
179 | break; | |
180 | } | |
181 | ||
182 | return prot; | |
183 | } | |
184 | ||
185 | static void *iova_to_va(int dev_fd, uint64_t iova, uint64_t *len) | |
186 | { | |
187 | int fd; | |
188 | void *addr; | |
189 | size_t size; | |
190 | struct vduse_iotlb_entry entry; | |
191 | ||
192 | entry.start = iova; | |
193 | entry.last = iova; | |
194 | ||
195 | /* | |
196 | * Find the first IOVA region that overlaps with the specified | |
197 | * range [start, last] and return the corresponding file descriptor. | |
198 | */ | |
199 | fd = ioctl(dev_fd, VDUSE_IOTLB_GET_FD, &entry); | |
200 | if (fd < 0) | |
201 | return NULL; | |
202 | ||
203 | size = entry.last - entry.start + 1; | |
204 | *len = entry.last - iova + 1; | |
205 | addr = mmap(0, size, perm_to_prot(entry.perm), MAP_SHARED, | |
206 | fd, entry.offset); | |
207 | close(fd); | |
208 | if (addr == MAP_FAILED) | |
209 | return NULL; | |
210 | ||
211 | /* | |
212 | * Using some data structures such as linked list to store | |
213 | * the iotlb mapping. The munmap(2) should be called for the | |
214 | * cached mapping when the corresponding VDUSE_UPDATE_IOTLB | |
215 | * message is received or the device is reset. | |
216 | */ | |
217 | ||
218 | return addr + iova - entry.start; | |
219 | } | |
220 | ||
221 | 3. Setup the kick eventfd for the specified virtqueues with the VDUSE_VQ_SETUP_KICKFD | |
222 | ioctl. The kick eventfd is used by VDUSE kernel module to notify userspace to | |
223 | consume the available ring. This is optional since userspace can choose to poll the | |
224 | available ring instead. | |
225 | ||
226 | 4. Listen to the kick eventfd (optional) and consume the available ring. The buffer | |
227 | described by the descriptors in the descriptor table should be also mapped into | |
228 | userspace via the VDUSE_IOTLB_GET_FD ioctl before accessing. | |
229 | ||
230 | 5. Inject an interrupt for specific virtqueue with the VDUSE_INJECT_VQ_IRQ ioctl | |
231 | after the used ring is filled. | |
232 | ||
233 | For more details on the uAPI, please see include/uapi/linux/vduse.h. |