Commit | Line | Data |
---|---|---|
a6546f89 | 1 | .. SPDX-License-Identifier: GPL-2.0-only |
2a26ed8e MCC |
2 | .. include:: <isonum.txt> |
3 | ||
4 | ===================== | |
5 | VFIO Mediated devices | |
6 | ===================== | |
7 | ||
8 | :Copyright: |copy| 2016, NVIDIA CORPORATION. All rights reserved. | |
9 | :Author: Neo Jia <cjia@nvidia.com> | |
10 | :Author: Kirti Wankhede <kwankhede@nvidia.com> | |
11 | ||
2a26ed8e | 12 | |
8e1c5a40 KW |
13 | |
14 | Virtual Function I/O (VFIO) Mediated devices[1] | |
15 | =============================================== | |
16 | ||
17 | The number of use cases for virtualizing DMA devices that do not have built-in | |
18 | SR_IOV capability is increasing. Previously, to virtualize such devices, | |
19 | developers had to create their own management interfaces and APIs, and then | |
20 | integrate them with user space software. To simplify integration with user space | |
21 | software, we have identified common requirements and a unified management | |
22 | interface for such devices. | |
23 | ||
24 | The VFIO driver framework provides unified APIs for direct device access. It is | |
25 | an IOMMU/device-agnostic framework for exposing direct device access to user | |
26 | space in a secure, IOMMU-protected environment. This framework is used for | |
27 | multiple devices, such as GPUs, network adapters, and compute accelerators. With | |
28 | direct device access, virtual machines or user space applications have direct | |
29 | access to the physical device. This framework is reused for mediated devices. | |
30 | ||
31 | The mediated core driver provides a common interface for mediated device | |
32 | management that can be used by drivers of different devices. This module | |
33 | provides a generic interface to perform these operations: | |
34 | ||
35 | * Create and destroy a mediated device | |
36 | * Add a mediated device to and remove it from a mediated bus driver | |
37 | * Add a mediated device to and remove it from an IOMMU group | |
38 | ||
39 | The mediated core driver also provides an interface to register a bus driver. | |
40 | For example, the mediated VFIO mdev driver is designed for mediated devices and | |
41 | supports VFIO APIs. The mediated bus driver adds a mediated device to and | |
42 | removes it from a VFIO group. | |
43 | ||
44 | The following high-level block diagram shows the main components and interfaces | |
45 | in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM | |
2a26ed8e | 46 | devices as examples, as these devices are the first devices to use this module:: |
8e1c5a40 KW |
47 | |
48 | +---------------+ | |
49 | | | | |
50 | | +-----------+ | mdev_register_driver() +--------------+ | |
51 | | | | +<------------------------+ | | |
52 | | | mdev | | | | | |
53 | | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user | |
54 | | | driver | | probe()/remove() | | APIs | |
55 | | | | | +--------------+ | |
56 | | +-----------+ | | |
57 | | | | |
58 | | MDEV CORE | | |
59 | | MODULE | | |
60 | | mdev.ko | | |
89345d51 | 61 | | +-----------+ | mdev_register_parent() +--------------+ |
8e1c5a40 | 62 | | | | +<------------------------+ | |
32328681 | 63 | | | | | | ccw_device.ko|<-> physical |
8e1c5a40 KW |
64 | | | | +------------------------>+ | device |
65 | | | | | callbacks +--------------+ | |
66 | | | Physical | | | |
89345d51 | 67 | | | device | | mdev_register_parent() +--------------+ |
8e1c5a40 KW |
68 | | | interface | |<------------------------+ | |
69 | | | | | | i915.ko |<-> physical | |
70 | | | | +------------------------>+ | device | |
71 | | | | | callbacks +--------------+ | |
8e1c5a40 KW |
72 | | +-----------+ | |
73 | +---------------+ | |
74 | ||
75 | ||
76 | Registration Interfaces | |
77 | ======================= | |
78 | ||
79 | The mediated core driver provides the following types of registration | |
80 | interfaces: | |
81 | ||
82 | * Registration interface for a mediated bus driver | |
83 | * Physical device driver interface | |
84 | ||
85 | Registration Interface for a Mediated Bus Driver | |
86 | ------------------------------------------------ | |
87 | ||
88a21f26 | 88 | The registration interface for a mediated device driver provides the following |
2a26ed8e | 89 | structure to represent a mediated device's driver:: |
8e1c5a40 KW |
90 | |
91 | /* | |
92 | * struct mdev_driver [2] - Mediated device's driver | |
8e1c5a40 KW |
93 | * @probe: called when new device created |
94 | * @remove: called when device removed | |
95 | * @driver: device driver structure | |
96 | */ | |
97 | struct mdev_driver { | |
2a3d15f2 JG |
98 | int (*probe) (struct mdev_device *dev); |
99 | void (*remove) (struct mdev_device *dev); | |
f2fbc72e | 100 | unsigned int (*get_available)(struct mdev_type *mtype); |
685a1537 | 101 | ssize_t (*show_description)(struct mdev_type *mtype, char *buf); |
8e1c5a40 KW |
102 | struct device_driver driver; |
103 | }; | |
104 | ||
105 | A mediated bus driver for mdev should use this structure in the function calls | |
106 | to register and unregister itself with the core driver: | |
107 | ||
2a26ed8e | 108 | * Register:: |
8e1c5a40 | 109 | |
d1877e63 | 110 | int mdev_register_driver(struct mdev_driver *drv); |
8e1c5a40 | 111 | |
2a26ed8e | 112 | * Unregister:: |
8e1c5a40 | 113 | |
d1877e63 | 114 | void mdev_unregister_driver(struct mdev_driver *drv); |
8e1c5a40 | 115 | |
6b42f491 JG |
116 | The mediated bus driver's probe function should create a vfio_device on top of |
117 | the mdev_device and connect it to an appropriate implementation of | |
118 | vfio_device_ops. | |
8e1c5a40 | 119 | |
88a21f26 JG |
120 | When a driver wants to add the GUID creation sysfs to an existing device it has |
121 | probe'd to then it should call:: | |
8e1c5a40 | 122 | |
89345d51 CH |
123 | int mdev_register_parent(struct mdev_parent *parent, struct device *dev, |
124 | struct mdev_driver *mdev_driver); | |
8e1c5a40 | 125 | |
88a21f26 JG |
126 | This will provide the 'mdev_supported_types/XX/create' files which can then be |
127 | used to trigger the creation of a mdev_device. The created mdev_device will be | |
128 | attached to the specified driver. | |
129 | ||
130 | When the driver needs to remove itself it calls:: | |
8e1c5a40 | 131 | |
89345d51 | 132 | void mdev_unregister_parent(struct mdev_parent *parent); |
8e1c5a40 | 133 | |
88a21f26 | 134 | Which will unbind and destroy all the created mdevs and remove the sysfs files. |
8e1c5a40 KW |
135 | |
136 | Mediated Device Management Interface Through sysfs | |
137 | ================================================== | |
138 | ||
139 | The management interface through sysfs enables user space software, such as | |
140 | libvirt, to query and configure mediated devices in a hardware-agnostic fashion. | |
141 | This management interface provides flexibility to the underlying physical | |
142 | device's driver to support features such as: | |
143 | ||
144 | * Mediated device hot plug | |
145 | * Multiple mediated devices in a single virtual machine | |
146 | * Multiple mediated devices from different physical devices | |
147 | ||
148 | Links in the mdev_bus Class Directory | |
149 | ------------------------------------- | |
150 | The /sys/class/mdev_bus/ directory contains links to devices that are registered | |
151 | with the mdev core driver. | |
152 | ||
153 | Directories and files under the sysfs for Each Physical Device | |
154 | -------------------------------------------------------------- | |
155 | ||
2a26ed8e MCC |
156 | :: |
157 | ||
158 | |- [parent physical device] | |
159 | |--- Vendor-specific-attributes [optional] | |
160 | |--- [mdev_supported_types] | |
161 | | |--- [<type-id>] | |
162 | | | |--- create | |
163 | | | |--- name | |
164 | | | |--- available_instances | |
165 | | | |--- device_api | |
166 | | | |--- description | |
167 | | | |--- [devices] | |
168 | | |--- [<type-id>] | |
169 | | | |--- create | |
170 | | | |--- name | |
171 | | | |--- available_instances | |
172 | | | |--- device_api | |
173 | | | |--- description | |
174 | | | |--- [devices] | |
175 | | |--- [<type-id>] | |
176 | | |--- create | |
177 | | |--- name | |
178 | | |--- available_instances | |
179 | | |--- device_api | |
180 | | |--- description | |
181 | | |--- [devices] | |
8e1c5a40 KW |
182 | |
183 | * [mdev_supported_types] | |
184 | ||
185 | The list of currently supported mediated device types and their details. | |
186 | ||
187 | [<type-id>], device_api, and available_instances are mandatory attributes | |
188 | that should be provided by vendor driver. | |
189 | ||
190 | * [<type-id>] | |
191 | ||
1c4f128e SD |
192 | The [<type-id>] name is created by adding the device driver string as a prefix |
193 | to the string provided by the vendor driver. This format of this name is as | |
2a26ed8e | 194 | follows:: |
8e1c5a40 KW |
195 | |
196 | sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); | |
197 | ||
198 | * device_api | |
199 | ||
290aac5d | 200 | This attribute shows which device API is being created, for example, |
8e1c5a40 KW |
201 | "vfio-pci" for a PCI device. |
202 | ||
203 | * available_instances | |
204 | ||
f2fbc72e | 205 | This attribute shows the number of devices of type <type-id> that can be |
8e1c5a40 KW |
206 | created. |
207 | ||
208 | * [device] | |
209 | ||
210 | This directory contains links to the devices of type <type-id> that have been | |
2a26ed8e | 211 | created. |
8e1c5a40 KW |
212 | |
213 | * name | |
214 | ||
0bc79069 | 215 | This attribute shows a human readable name. |
8e1c5a40 KW |
216 | |
217 | * description | |
218 | ||
685a1537 | 219 | This attribute can show brief features/description of the type. This is an |
8e1c5a40 KW |
220 | optional attribute. |
221 | ||
222 | Directories and Files Under the sysfs for Each mdev Device | |
223 | ---------------------------------------------------------- | |
224 | ||
2a26ed8e MCC |
225 | :: |
226 | ||
227 | |- [parent phy device] | |
228 | |--- [$MDEV_UUID] | |
8e1c5a40 KW |
229 | |--- remove |
230 | |--- mdev_type {link to its type} | |
231 | |--- vendor-specific-attributes [optional] | |
232 | ||
233 | * remove (write only) | |
2a26ed8e | 234 | |
8e1c5a40 KW |
235 | Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can |
236 | fail the remove() callback if that device is active and the vendor driver | |
237 | doesn't support hot unplug. | |
238 | ||
2a26ed8e MCC |
239 | Example:: |
240 | ||
8e1c5a40 KW |
241 | # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove |
242 | ||
2a26ed8e | 243 | Mediated device Hot plug |
8e1c5a40 KW |
244 | ------------------------ |
245 | ||
246 | Mediated devices can be created and assigned at runtime. The procedure to hot | |
247 | plug a mediated device is the same as the procedure to hot plug a PCI device. | |
248 | ||
249 | Translation APIs for Mediated Devices | |
250 | ===================================== | |
251 | ||
252 | The following APIs are provided for translating user pfn to host pfn in a VFIO | |
2a26ed8e | 253 | driver:: |
8e1c5a40 | 254 | |
44abdd16 | 255 | int vfio_pin_pages(struct vfio_device *device, dma_addr_t iova, |
34a255e6 | 256 | int npage, int prot, struct page **pages); |
8e1c5a40 | 257 | |
44abdd16 | 258 | void vfio_unpin_pages(struct vfio_device *device, dma_addr_t iova, |
2a26ed8e | 259 | int npage); |
8e1c5a40 KW |
260 | |
261 | These functions call back into the back-end IOMMU module by using the pin_pages | |
262 | and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently | |
263 | these callbacks are supported in the TYPE1 IOMMU module. To enable them for | |
264 | other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide | |
265 | these two callback functions. | |
266 | ||
267 | References | |
9d1a546c | 268 | ========== |
8e1c5a40 | 269 | |
baa293e9 | 270 | 1. See Documentation/driver-api/vfio.rst for more information on VFIO. |
2a26ed8e MCC |
271 | 2. struct mdev_driver in include/linux/mdev.h |
272 | 3. struct mdev_parent_ops in include/linux/mdev.h | |
273 | 4. struct vfio_iommu_driver_ops in include/linux/vfio.h |