Commit | Line | Data |
---|---|---|
492a6be1 TK |
1 | Introduction: |
2 | ============ | |
3 | The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised | |
4 | of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards. | |
5 | The AP devices provide cryptographic functions to all CPUs assigned to a | |
6 | linux system running in an IBM Z system LPAR. | |
7 | ||
8 | The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap | |
9 | is to make AP cards available to KVM guests using the VFIO mediated device | |
10 | framework. This implementation relies considerably on the s390 virtualization | |
11 | facilities which do most of the hard work of providing direct access to AP | |
12 | devices. | |
13 | ||
14 | AP Architectural Overview: | |
15 | ========================= | |
16 | To facilitate the comprehension of the design, let's start with some | |
17 | definitions: | |
18 | ||
19 | * AP adapter | |
20 | ||
21 | An AP adapter is an IBM Z adapter card that can perform cryptographic | |
22 | functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters | |
23 | assigned to the LPAR in which a linux host is running will be available to | |
24 | the linux host. Each adapter is identified by a number from 0 to 255; however, | |
25 | the maximum adapter number is determined by machine model and/or adapter type. | |
26 | When installed, an AP adapter is accessed by AP instructions executed by any | |
27 | CPU. | |
28 | ||
29 | The AP adapter cards are assigned to a given LPAR via the system's Activation | |
30 | Profile which can be edited via the HMC. When the linux host system is IPL'd | |
31 | in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and | |
32 | creates a sysfs device for each assigned adapter. For example, if AP adapters | |
33 | 4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following | |
34 | sysfs device entries: | |
35 | ||
36 | /sys/devices/ap/card04 | |
37 | /sys/devices/ap/card0a | |
38 | ||
39 | Symbolic links to these devices will also be created in the AP bus devices | |
40 | sub-directory: | |
41 | ||
42 | /sys/bus/ap/devices/[card04] | |
43 | /sys/bus/ap/devices/[card04] | |
44 | ||
45 | * AP domain | |
46 | ||
47 | An adapter is partitioned into domains. An adapter can hold up to 256 domains | |
48 | depending upon the adapter type and hardware configuration. A domain is | |
49 | identified by a number from 0 to 255; however, the maximum domain number is | |
50 | determined by machine model and/or adapter type.. A domain can be thought of | |
51 | as a set of hardware registers and memory used for processing AP commands. A | |
52 | domain can be configured with a secure private key used for clear key | |
53 | encryption. A domain is classified in one of two ways depending upon how it | |
54 | may be accessed: | |
55 | ||
56 | * Usage domains are domains that are targeted by an AP instruction to | |
57 | process an AP command. | |
58 | ||
59 | * Control domains are domains that are changed by an AP command sent to a | |
60 | usage domain; for example, to set the secure private key for the control | |
61 | domain. | |
62 | ||
63 | The AP usage and control domains are assigned to a given LPAR via the system's | |
64 | Activation Profile which can be edited via the HMC. When a linux host system | |
65 | is IPL'd in the LPAR, the AP bus module detects the AP usage and control | |
66 | domains assigned to the LPAR. The domain number of each usage domain and | |
67 | adapter number of each AP adapter are combined to create AP queue devices | |
68 | (see AP Queue section below). The domain number of each control domain will be | |
69 | represented in a bitmask and stored in a sysfs file | |
70 | /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least | |
71 | significant bit, correspond to domains 0-255. | |
72 | ||
73 | * AP Queue | |
74 | ||
75 | An AP queue is the means by which an AP command is sent to a usage domain | |
76 | inside a specific adapter. An AP queue is identified by a tuple | |
77 | comprised of an AP adapter ID (APID) and an AP queue index (APQI). The | |
78 | APQI corresponds to a given usage domain number within the adapter. This tuple | |
79 | forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP | |
80 | instructions include a field containing the APQN to identify the AP queue to | |
81 | which the AP command is to be sent for processing. | |
82 | ||
83 | The AP bus will create a sysfs device for each APQN that can be derived from | |
84 | the cross product of the AP adapter and usage domain numbers detected when the | |
85 | AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage | |
86 | domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the | |
87 | following sysfs entries: | |
88 | ||
89 | /sys/devices/ap/card04/04.0006 | |
90 | /sys/devices/ap/card04/04.0047 | |
91 | /sys/devices/ap/card0a/0a.0006 | |
92 | /sys/devices/ap/card0a/0a.0047 | |
93 | ||
94 | The following symbolic links to these devices will be created in the AP bus | |
95 | devices subdirectory: | |
96 | ||
97 | /sys/bus/ap/devices/[04.0006] | |
98 | /sys/bus/ap/devices/[04.0047] | |
99 | /sys/bus/ap/devices/[0a.0006] | |
100 | /sys/bus/ap/devices/[0a.0047] | |
101 | ||
102 | * AP Instructions: | |
103 | ||
104 | There are three AP instructions: | |
105 | ||
106 | * NQAP: to enqueue an AP command-request message to a queue | |
107 | * DQAP: to dequeue an AP command-reply message from a queue | |
108 | * PQAP: to administer the queues | |
109 | ||
110 | AP instructions identify the domain that is targeted to process the AP | |
111 | command; this must be one of the usage domains. An AP command may modify a | |
112 | domain that is not one of the usage domains, but the modified domain | |
113 | must be one of the control domains. | |
114 | ||
115 | AP and SIE: | |
116 | ========== | |
117 | Let's now take a look at how AP instructions executed on a guest are interpreted | |
118 | by the hardware. | |
119 | ||
120 | A satellite control block called the Crypto Control Block (CRYCB) is attached to | |
121 | our main hardware virtualization control block. The CRYCB contains three fields | |
122 | to identify the adapters, usage domains and control domains assigned to the KVM | |
123 | guest: | |
124 | ||
125 | * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned | |
126 | to the KVM guest. Each bit in the mask, from left to right (i.e. from most | |
127 | significant to least significant bit in big endian order), corresponds to | |
128 | an APID from 0-255. If a bit is set, the corresponding adapter is valid for | |
129 | use by the KVM guest. | |
130 | ||
131 | * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains | |
132 | assigned to the KVM guest. Each bit in the mask, from left to right (i.e. from | |
133 | most significant to least significant bit in big endian order), corresponds to | |
134 | an AP queue index (APQI) from 0-255. If a bit is set, the corresponding queue | |
135 | is valid for use by the KVM guest. | |
136 | ||
137 | * The AP Domain Mask field is a bit mask that identifies the AP control domains | |
138 | assigned to the KVM guest. The ADM bit mask controls which domains can be | |
139 | changed by an AP command-request message sent to a usage domain from the | |
140 | guest. Each bit in the mask, from left to right (i.e. from most significant to | |
141 | least significant bit in big endian order), corresponds to a domain from | |
142 | 0-255. If a bit is set, the corresponding domain can be modified by an AP | |
143 | command-request message sent to a usage domain. | |
144 | ||
145 | If you recall from the description of an AP Queue, AP instructions include | |
146 | an APQN to identify the AP queue to which an AP command-request message is to be | |
147 | sent (NQAP and PQAP instructions), or from which a command-reply message is to | |
148 | be received (DQAP instruction). The validity of an APQN is defined by the matrix | |
149 | calculated from the APM and AQM; it is the cross product of all assigned adapter | |
150 | numbers (APM) with all assigned queue indexes (AQM). For example, if adapters 1 | |
151 | and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs (1,5), (1,6), | |
152 | (2,5) and (2,6) will be valid for the guest. | |
153 | ||
154 | The APQNs can provide secure key functionality - i.e., a private key is stored | |
155 | on the adapter card for each of its domains - so each APQN must be assigned to | |
156 | at most one guest or to the linux host. | |
157 | ||
158 | Example 1: Valid configuration: | |
159 | ------------------------------ | |
160 | Guest1: adapters 1,2 domains 5,6 | |
161 | Guest2: adapter 1,2 domain 7 | |
162 | ||
163 | This is valid because both guests have a unique set of APQNs: | |
164 | Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); | |
165 | Guest2 has APQNs (1,7), (2,7) | |
166 | ||
167 | Example 2: Valid configuration: | |
168 | ------------------------------ | |
169 | Guest1: adapters 1,2 domains 5,6 | |
170 | Guest2: adapters 3,4 domains 5,6 | |
171 | ||
172 | This is also valid because both guests have a unique set of APQNs: | |
173 | Guest1 has APQNs (1,5), (1,6), (2,5), (2,6); | |
174 | Guest2 has APQNs (3,5), (3,6), (4,5), (4,6) | |
175 | ||
176 | Example 3: Invalid configuration: | |
177 | -------------------------------- | |
178 | Guest1: adapters 1,2 domains 5,6 | |
179 | Guest2: adapter 1 domains 6,7 | |
180 | ||
181 | This is an invalid configuration because both guests have access to | |
182 | APQN (1,6). | |
183 | ||
184 | The Design: | |
185 | =========== | |
186 | The design introduces three new objects: | |
187 | ||
188 | 1. AP matrix device | |
189 | 2. VFIO AP device driver (vfio_ap.ko) | |
190 | 3. VFIO AP mediated matrix pass-through device | |
191 | ||
192 | The VFIO AP device driver | |
193 | ------------------------- | |
194 | The VFIO AP (vfio_ap) device driver serves the following purposes: | |
195 | ||
196 | 1. Provides the interfaces to secure APQNs for exclusive use of KVM guests. | |
197 | ||
198 | 2. Sets up the VFIO mediated device interfaces to manage a mediated matrix | |
199 | device and creates the sysfs interfaces for assigning adapters, usage | |
200 | domains, and control domains comprising the matrix for a KVM guest. | |
201 | ||
202 | 3. Configures the APM, AQM and ADM in the CRYCB referenced by a KVM guest's | |
203 | SIE state description to grant the guest access to a matrix of AP devices | |
204 | ||
205 | Reserve APQNs for exclusive use of KVM guests | |
206 | --------------------------------------------- | |
207 | The following block diagram illustrates the mechanism by which APQNs are | |
208 | reserved: | |
209 | ||
210 | +------------------+ | |
211 | 7 remove | | | |
212 | +--------------------> cex4queue driver | | |
213 | | | | | |
214 | | +------------------+ | |
215 | | | |
216 | | | |
217 | | +------------------+ +-----------------+ | |
218 | | 5 register driver | | 3 create | | | |
219 | | +----------------> Device core +----------> matrix device | | |
220 | | | | | | | | |
221 | | | +--------^---------+ +-----------------+ | |
222 | | | | | |
223 | | | +-------------------+ | |
224 | | | +-----------------------------------+ | | |
225 | | | | 4 register AP driver | | 2 register device | |
226 | | | | | | | |
227 | +--------+---+-v---+ +--------+-------+-+ | |
228 | | | | | | |
229 | | ap_bus +--------------------- > vfio_ap driver | | |
230 | | | 8 probe | | | |
231 | +--------^---------+ +--^--^------------+ | |
232 | 6 edit | | | | |
233 | apmask | +-----------------------------+ | 9 mdev create | |
234 | aqmask | | 1 modprobe | | |
235 | +--------+-----+---+ +----------------+-+ +------------------+ | |
236 | | | | |8 create | mediated | | |
237 | | admin | | VFIO device core |---------> matrix | | |
238 | | + | | | device | | |
239 | +------+-+---------+ +--------^---------+ +--------^---------+ | |
240 | | | | | | |
241 | | | 9 create vfio_ap-passthrough | | | |
242 | | +------------------------------+ | | |
243 | +-------------------------------------------------------------+ | |
244 | 10 assign adapter/domain/control domain | |
245 | ||
246 | The process for reserving an AP queue for use by a KVM guest is: | |
247 | ||
248 | 1. The administrator loads the vfio_ap device driver | |
249 | 2. The vfio-ap driver during its initialization will register a single 'matrix' | |
250 | device with the device core. This will serve as the parent device for | |
251 | all mediated matrix devices used to configure an AP matrix for a guest. | |
252 | 3. The /sys/devices/vfio_ap/matrix device is created by the device core | |
253 | 4 The vfio_ap device driver will register with the AP bus for AP queue devices | |
254 | of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap | |
255 | driver's probe and remove callback interfaces. Devices older than CEX4 queues | |
256 | are not supported to simplify the implementation by not needlessly | |
257 | complicating the design by supporting older devices that will go out of | |
258 | service in the relatively near future, and for which there are few older | |
259 | systems around on which to test. | |
260 | 5. The AP bus registers the vfio_ap device driver with the device core | |
261 | 6. The administrator edits the AP adapter and queue masks to reserve AP queues | |
262 | for use by the vfio_ap device driver. | |
263 | 7. The AP bus removes the AP queues reserved for the vfio_ap driver from the | |
264 | default zcrypt cex4queue driver. | |
265 | 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for | |
266 | it. | |
267 | 9. The administrator creates a passthrough type mediated matrix device to be | |
268 | used by a guest | |
269 | 10 The administrator assigns the adapters, usage domains and control domains | |
270 | to be exclusively used by a guest. | |
271 | ||
272 | Set up the VFIO mediated device interfaces | |
273 | ------------------------------------------ | |
274 | The VFIO AP device driver utilizes the common interface of the VFIO mediated | |
275 | device core driver to: | |
276 | * Register an AP mediated bus driver to add a mediated matrix device to and | |
277 | remove it from a VFIO group. | |
278 | * Create and destroy a mediated matrix device | |
279 | * Add a mediated matrix device to and remove it from the AP mediated bus driver | |
280 | * Add a mediated matrix device to and remove it from an IOMMU group | |
281 | ||
282 | The following high-level block diagram shows the main components and interfaces | |
283 | of the VFIO AP mediated matrix device driver: | |
284 | ||
285 | +-------------+ | |
286 | | | | |
287 | | +---------+ | mdev_register_driver() +--------------+ | |
288 | | | Mdev | +<-----------------------+ | | |
289 | | | bus | | | vfio_mdev.ko | | |
290 | | | driver | +----------------------->+ |<-> VFIO user | |
291 | | +---------+ | probe()/remove() +--------------+ APIs | |
292 | | | | |
293 | | MDEV CORE | | |
294 | | MODULE | | |
295 | | mdev.ko | | |
296 | | +---------+ | mdev_register_device() +--------------+ | |
297 | | |Physical | +<-----------------------+ | | |
298 | | | device | | | vfio_ap.ko |<-> matrix | |
299 | | |interface| +----------------------->+ | device | |
300 | | +---------+ | callback +--------------+ | |
301 | +-------------+ | |
302 | ||
303 | During initialization of the vfio_ap module, the matrix device is registered | |
304 | with an 'mdev_parent_ops' structure that provides the sysfs attribute | |
305 | structures, mdev functions and callback interfaces for managing the mediated | |
306 | matrix device. | |
307 | ||
308 | * sysfs attribute structures: | |
309 | * supported_type_groups | |
310 | The VFIO mediated device framework supports creation of user-defined | |
311 | mediated device types. These mediated device types are specified | |
312 | via the 'supported_type_groups' structure when a device is registered | |
313 | with the mediated device framework. The registration process creates the | |
314 | sysfs structures for each mediated device type specified in the | |
315 | 'mdev_supported_types' sub-directory of the device being registered. Along | |
316 | with the device type, the sysfs attributes of the mediated device type are | |
317 | provided. | |
318 | ||
319 | The VFIO AP device driver will register one mediated device type for | |
320 | passthrough devices: | |
321 | /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough | |
322 | Only the read-only attributes required by the VFIO mdev framework will | |
323 | be provided: | |
324 | ... name | |
325 | ... device_api | |
326 | ... available_instances | |
327 | ... device_api | |
328 | Where: | |
329 | * name: specifies the name of the mediated device type | |
330 | * device_api: the mediated device type's API | |
331 | * available_instances: the number of mediated matrix passthrough devices | |
332 | that can be created | |
333 | * device_api: specifies the VFIO API | |
334 | * mdev_attr_groups | |
335 | This attribute group identifies the user-defined sysfs attributes of the | |
336 | mediated device. When a device is registered with the VFIO mediated device | |
337 | framework, the sysfs attribute files identified in the 'mdev_attr_groups' | |
338 | structure will be created in the mediated matrix device's directory. The | |
339 | sysfs attributes for a mediated matrix device are: | |
340 | * assign_adapter: | |
341 | * unassign_adapter: | |
342 | Write-only attributes for assigning/unassigning an AP adapter to/from the | |
343 | mediated matrix device. To assign/unassign an adapter, the APID of the | |
344 | adapter is echoed to the respective attribute file. | |
345 | * assign_domain: | |
346 | * unassign_domain: | |
347 | Write-only attributes for assigning/unassigning an AP usage domain to/from | |
348 | the mediated matrix device. To assign/unassign a domain, the domain | |
349 | number of the the usage domain is echoed to the respective attribute | |
350 | file. | |
351 | * matrix: | |
352 | A read-only file for displaying the APQNs derived from the cross product | |
353 | of the adapter and domain numbers assigned to the mediated matrix device. | |
354 | * assign_control_domain: | |
355 | * unassign_control_domain: | |
356 | Write-only attributes for assigning/unassigning an AP control domain | |
357 | to/from the mediated matrix device. To assign/unassign a control domain, | |
358 | the ID of the domain to be assigned/unassigned is echoed to the respective | |
359 | attribute file. | |
360 | * control_domains: | |
361 | A read-only file for displaying the control domain numbers assigned to the | |
362 | mediated matrix device. | |
363 | ||
364 | * functions: | |
365 | * create: | |
366 | allocates the ap_matrix_mdev structure used by the vfio_ap driver to: | |
367 | * Store the reference to the KVM structure for the guest using the mdev | |
368 | * Store the AP matrix configuration for the adapters, domains, and control | |
369 | domains assigned via the corresponding sysfs attributes files | |
370 | * remove: | |
371 | deallocates the mediated matrix device's ap_matrix_mdev structure. This will | |
372 | be allowed only if a running guest is not using the mdev. | |
373 | ||
374 | * callback interfaces | |
375 | * open: | |
376 | The vfio_ap driver uses this callback to register a | |
377 | VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the mdev matrix | |
378 | device. The open is invoked when QEMU connects the VFIO iommu group | |
379 | for the mdev matrix device to the MDEV bus. Access to the KVM structure used | |
380 | to configure the KVM guest is provided via this callback. The KVM structure, | |
381 | is used to configure the guest's access to the AP matrix defined via the | |
382 | mediated matrix device's sysfs attribute files. | |
383 | * release: | |
384 | unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the | |
385 | mdev matrix device and deconfigures the guest's AP matrix. | |
386 | ||
387 | Configure the APM, AQM and ADM in the CRYCB: | |
388 | ------------------------------------------- | |
389 | Configuring the AP matrix for a KVM guest will be performed when the | |
390 | VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier | |
391 | function is called when QEMU connects to KVM. The guest's AP matrix is | |
392 | configured via it's CRYCB by: | |
393 | * Setting the bits in the APM corresponding to the APIDs assigned to the | |
394 | mediated matrix device via its 'assign_adapter' interface. | |
395 | * Setting the bits in the AQM corresponding to the domains assigned to the | |
396 | mediated matrix device via its 'assign_domain' interface. | |
397 | * Setting the bits in the ADM corresponding to the domain dIDs assigned to the | |
398 | mediated matrix device via its 'assign_control_domains' interface. | |
399 | ||
400 | The CPU model features for AP | |
401 | ----------------------------- | |
402 | The AP stack relies on the presence of the AP instructions as well as two | |
403 | facilities: The AP Facilities Test (APFT) facility; and the AP Query | |
404 | Configuration Information (QCI) facility. These features/facilities are made | |
405 | available to a KVM guest via the following CPU model features: | |
406 | ||
407 | 1. ap: Indicates whether the AP instructions are installed on the guest. This | |
408 | feature will be enabled by KVM only if the AP instructions are installed | |
409 | on the host. | |
410 | ||
411 | 2. apft: Indicates the APFT facility is available on the guest. This facility | |
412 | can be made available to the guest only if it is available on the host (i.e., | |
413 | facility bit 15 is set). | |
414 | ||
415 | 3. apqci: Indicates the AP QCI facility is available on the guest. This facility | |
416 | can be made available to the guest only if it is available on the host (i.e., | |
417 | facility bit 12 is set). | |
418 | ||
419 | Note: If the user chooses to specify a CPU model different than the 'host' | |
420 | model to QEMU, the CPU model features and facilities need to be turned on | |
421 | explicitly; for example: | |
422 | ||
423 | /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on | |
424 | ||
425 | A guest can be precluded from using AP features/facilities by turning them off | |
426 | explicitly; for example: | |
427 | ||
428 | /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off | |
429 | ||
430 | Note: If the APFT facility is turned off (apft=off) for the guest, the guest | |
431 | will not see any AP devices. The zcrypt device drivers that register for type 10 | |
432 | and newer AP devices - i.e., the cex4card and cex4queue device drivers - need | |
433 | the APFT facility to ascertain the facilities installed on a given AP device. If | |
434 | the APFT facility is not installed on the guest, then the probe of device | |
435 | drivers will fail since only type 10 and newer devices can be configured for | |
436 | guest use. | |
437 | ||
438 | Example: | |
439 | ======= | |
440 | Let's now provide an example to illustrate how KVM guests may be given | |
441 | access to AP facilities. For this example, we will show how to configure | |
442 | three guests such that executing the lszcrypt command on the guests would | |
443 | look like this: | |
444 | ||
445 | Guest1 | |
446 | ------ | |
447 | CARD.DOMAIN TYPE MODE | |
448 | ------------------------------ | |
449 | 05 CEX5C CCA-Coproc | |
450 | 05.0004 CEX5C CCA-Coproc | |
451 | 05.00ab CEX5C CCA-Coproc | |
452 | 06 CEX5A Accelerator | |
453 | 06.0004 CEX5A Accelerator | |
454 | 06.00ab CEX5C CCA-Coproc | |
455 | ||
456 | Guest2 | |
457 | ------ | |
458 | CARD.DOMAIN TYPE MODE | |
459 | ------------------------------ | |
460 | 05 CEX5A Accelerator | |
461 | 05.0047 CEX5A Accelerator | |
462 | 05.00ff CEX5A Accelerator | |
463 | ||
464 | Guest2 | |
465 | ------ | |
466 | CARD.DOMAIN TYPE MODE | |
467 | ------------------------------ | |
468 | 06 CEX5A Accelerator | |
469 | 06.0047 CEX5A Accelerator | |
470 | 06.00ff CEX5A Accelerator | |
471 | ||
472 | These are the steps: | |
473 | ||
474 | 1. Install the vfio_ap module on the linux host. The dependency chain for the | |
475 | vfio_ap module is: | |
476 | * iommu | |
477 | * s390 | |
478 | * zcrypt | |
479 | * vfio | |
480 | * vfio_mdev | |
481 | * vfio_mdev_device | |
482 | * KVM | |
483 | ||
484 | To build the vfio_ap module, the kernel build must be configured with the | |
485 | following Kconfig elements selected: | |
486 | * IOMMU_SUPPORT | |
487 | * S390 | |
488 | * ZCRYPT | |
489 | * S390_AP_IOMMU | |
490 | * VFIO | |
491 | * VFIO_MDEV | |
492 | * VFIO_MDEV_DEVICE | |
493 | * KVM | |
494 | ||
495 | If using make menuconfig select the following to build the vfio_ap module: | |
496 | -> Device Drivers | |
497 | -> IOMMU Hardware Support | |
498 | select S390 AP IOMMU Support | |
499 | -> VFIO Non-Privileged userspace driver framework | |
500 | -> Mediated device driver frramework | |
501 | -> VFIO driver for Mediated devices | |
502 | -> I/O subsystem | |
503 | -> VFIO support for AP devices | |
504 | ||
505 | 2. Secure the AP queues to be used by the three guests so that the host can not | |
506 | access them. To secure them, there are two sysfs files that specify | |
507 | bitmasks marking a subset of the APQN range as 'usable by the default AP | |
508 | queue device drivers' or 'not usable by the default device drivers' and thus | |
509 | available for use by the vfio_ap device driver'. The location of the sysfs | |
510 | files containing the masks are: | |
511 | ||
512 | /sys/bus/ap/apmask | |
513 | /sys/bus/ap/aqmask | |
514 | ||
515 | The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs | |
516 | (APID). Each bit in the mask, from left to right (i.e., from most significant | |
517 | to least significant bit in big endian order), corresponds to an APID from | |
518 | 0-255. If a bit is set, the APID is marked as usable only by the default AP | |
519 | queue device drivers; otherwise, the APID is usable by the vfio_ap | |
520 | device driver. | |
521 | ||
522 | The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes | |
523 | (APQI). Each bit in the mask, from left to right (i.e., from most significant | |
524 | to least significant bit in big endian order), corresponds to an APQI from | |
525 | 0-255. If a bit is set, the APQI is marked as usable only by the default AP | |
526 | queue device drivers; otherwise, the APQI is usable by the vfio_ap device | |
527 | driver. | |
528 | ||
529 | Take, for example, the following mask: | |
530 | ||
531 | 0x7dffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff | |
532 | ||
533 | It indicates: | |
534 | ||
535 | 1, 2, 3, 4, 5, and 7-255 belong to the default drivers' pool, and 0 and 6 | |
536 | belong to the vfio_ap device driver's pool. | |
537 | ||
538 | The APQN of each AP queue device assigned to the linux host is checked by the | |
539 | AP bus against the set of APQNs derived from the cross product of APIDs | |
540 | and APQIs marked as usable only by the default AP queue device drivers. If a | |
541 | match is detected, only the default AP queue device drivers will be probed; | |
542 | otherwise, the vfio_ap device driver will be probed. | |
543 | ||
544 | By default, the two masks are set to reserve all APQNs for use by the default | |
545 | AP queue device drivers. There are two ways the default masks can be changed: | |
546 | ||
547 | 1. The sysfs mask files can be edited by echoing a string into the | |
548 | respective sysfs mask file in one of two formats: | |
549 | ||
550 | * An absolute hex string starting with 0x - like "0x12345678" - sets | |
551 | the mask. If the given string is shorter than the mask, it is padded | |
552 | with 0s on the right; for example, specifying a mask value of 0x41 is | |
553 | the same as specifying: | |
554 | ||
555 | 0x4100000000000000000000000000000000000000000000000000000000000000 | |
556 | ||
557 | Keep in mind that the mask reads from left to right (i.e., most | |
558 | significant to least significant bit in big endian order), so the mask | |
559 | above identifies device numbers 1 and 7 (01000001). | |
560 | ||
561 | If the string is longer than the mask, the operation is terminated with | |
562 | an error (EINVAL). | |
563 | ||
564 | * Individual bits in the mask can be switched on and off by specifying | |
565 | each bit number to be switched in a comma separated list. Each bit | |
566 | number string must be prepended with a ('+') or minus ('-') to indicate | |
567 | the corresponding bit is to be switched on ('+') or off ('-'). Some | |
568 | valid values are: | |
569 | ||
570 | "+0" switches bit 0 on | |
571 | "-13" switches bit 13 off | |
572 | "+0x41" switches bit 65 on | |
573 | "-0xff" switches bit 255 off | |
574 | ||
575 | The following example: | |
576 | +0,-6,+0x47,-0xf0 | |
577 | ||
578 | Switches bits 0 and 71 (0x47) on | |
579 | Switches bits 6 and 240 (0xf0) off | |
580 | ||
581 | Note that the bits not specified in the list remain as they were before | |
582 | the operation. | |
583 | ||
584 | 2. The masks can also be changed at boot time via parameters on the kernel | |
585 | command line like this: | |
586 | ||
587 | ap.apmask=0xffff ap.aqmask=0x40 | |
588 | ||
589 | This would create the following masks: | |
590 | ||
591 | apmask: | |
592 | 0xffff000000000000000000000000000000000000000000000000000000000000 | |
593 | ||
594 | aqmask: | |
595 | 0x4000000000000000000000000000000000000000000000000000000000000000 | |
596 | ||
597 | Resulting in these two pools: | |
598 | ||
599 | default drivers pool: adapter 0-15, domain 1 | |
600 | alternate drivers pool: adapter 16-255, domains 0, 2-255 | |
601 | ||
602 | Securing the APQNs for our example: | |
603 | ---------------------------------- | |
604 | To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047, | |
605 | 06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding | |
606 | APQNs can either be removed from the default masks: | |
607 | ||
608 | echo -5,-6 > /sys/bus/ap/apmask | |
609 | ||
610 | echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask | |
611 | ||
612 | Or the masks can be set as follows: | |
613 | ||
614 | echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \ | |
615 | > apmask | |
616 | ||
617 | echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \ | |
618 | > aqmask | |
619 | ||
620 | This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, | |
621 | 06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The | |
622 | sysfs directory for the vfio_ap device driver will now contain symbolic links | |
623 | to the AP queue devices bound to it: | |
624 | ||
625 | /sys/bus/ap | |
626 | ... [drivers] | |
627 | ...... [vfio_ap] | |
628 | ......... [05.0004] | |
629 | ......... [05.0047] | |
630 | ......... [05.00ab] | |
631 | ......... [05.00ff] | |
632 | ......... [06.0004] | |
633 | ......... [06.0047] | |
634 | ......... [06.00ab] | |
635 | ......... [06.00ff] | |
636 | ||
637 | Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later) | |
638 | can be bound to the vfio_ap device driver. The reason for this is to | |
639 | simplify the implementation by not needlessly complicating the design by | |
640 | supporting older devices that will go out of service in the relatively near | |
641 | future and for which there are few older systems on which to test. | |
642 | ||
643 | The administrator, therefore, must take care to secure only AP queues that | |
644 | can be bound to the vfio_ap device driver. The device type for a given AP | |
645 | queue device can be read from the parent card's sysfs directory. For example, | |
646 | to see the hardware type of the queue 05.0004: | |
647 | ||
648 | cat /sys/bus/ap/devices/card05/hwtype | |
649 | ||
650 | The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the | |
651 | vfio_ap device driver. | |
652 | ||
653 | 3. Create the mediated devices needed to configure the AP matrixes for the | |
654 | three guests and to provide an interface to the vfio_ap driver for | |
655 | use by the guests: | |
656 | ||
657 | /sys/devices/vfio_ap/matrix/ | |
658 | --- [mdev_supported_types] | |
659 | ------ [vfio_ap-passthrough] (passthrough mediated matrix device type) | |
660 | --------- create | |
661 | --------- [devices] | |
662 | ||
663 | To create the mediated devices for the three guests: | |
664 | ||
665 | uuidgen > create | |
666 | uuidgen > create | |
667 | uuidgen > create | |
668 | ||
669 | or | |
670 | ||
671 | echo $uuid1 > create | |
672 | echo $uuid2 > create | |
673 | echo $uuid3 > create | |
674 | ||
675 | This will create three mediated devices in the [devices] subdirectory named | |
676 | after the UUID written to the create attribute file. We call them $uuid1, | |
677 | $uuid2 and $uuid3 and this is the sysfs directory structure after creation: | |
678 | ||
679 | /sys/devices/vfio_ap/matrix/ | |
680 | --- [mdev_supported_types] | |
681 | ------ [vfio_ap-passthrough] | |
682 | --------- [devices] | |
683 | ------------ [$uuid1] | |
684 | --------------- assign_adapter | |
685 | --------------- assign_control_domain | |
686 | --------------- assign_domain | |
687 | --------------- matrix | |
688 | --------------- unassign_adapter | |
689 | --------------- unassign_control_domain | |
690 | --------------- unassign_domain | |
691 | ||
692 | ------------ [$uuid2] | |
693 | --------------- assign_adapter | |
694 | --------------- assign_control_domain | |
695 | --------------- assign_domain | |
696 | --------------- matrix | |
697 | --------------- unassign_adapter | |
698 | ----------------unassign_control_domain | |
699 | ----------------unassign_domain | |
700 | ||
701 | ------------ [$uuid3] | |
702 | --------------- assign_adapter | |
703 | --------------- assign_control_domain | |
704 | --------------- assign_domain | |
705 | --------------- matrix | |
706 | --------------- unassign_adapter | |
707 | ----------------unassign_control_domain | |
708 | ----------------unassign_domain | |
709 | ||
710 | 4. The administrator now needs to configure the matrixes for the mediated | |
711 | devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3). | |
712 | ||
713 | This is how the matrix is configured for Guest1: | |
714 | ||
715 | echo 5 > assign_adapter | |
716 | echo 6 > assign_adapter | |
717 | echo 4 > assign_domain | |
718 | echo 0xab > assign_domain | |
719 | ||
720 | Control domains can similarly be assigned using the assign_control_domain | |
721 | sysfs file. | |
722 | ||
723 | If a mistake is made configuring an adapter, domain or control domain, | |
724 | you can use the unassign_xxx files to unassign the adapter, domain or | |
725 | control domain. | |
726 | ||
727 | To display the matrix configuration for Guest1: | |
728 | ||
729 | cat matrix | |
730 | ||
731 | This is how the matrix is configured for Guest2: | |
732 | ||
733 | echo 5 > assign_adapter | |
734 | echo 0x47 > assign_domain | |
735 | echo 0xff > assign_domain | |
736 | ||
737 | This is how the matrix is configured for Guest3: | |
738 | ||
739 | echo 6 > assign_adapter | |
740 | echo 0x47 > assign_domain | |
741 | echo 0xff > assign_domain | |
742 | ||
743 | In order to successfully assign an adapter: | |
744 | ||
745 | * The adapter number specified must represent a value from 0 up to the | |
746 | maximum adapter number configured for the system. If an adapter number | |
747 | higher than the maximum is specified, the operation will terminate with | |
748 | an error (ENODEV). | |
749 | ||
750 | * All APQNs that can be derived from the adapter ID and the IDs of | |
751 | the previously assigned domains must be bound to the vfio_ap device | |
752 | driver. If no domains have yet been assigned, then there must be at least | |
753 | one APQN with the specified APID bound to the vfio_ap driver. If no such | |
754 | APQNs are bound to the driver, the operation will terminate with an | |
755 | error (EADDRNOTAVAIL). | |
756 | ||
757 | No APQN that can be derived from the adapter ID and the IDs of the | |
758 | previously assigned domains can be assigned to another mediated matrix | |
759 | device. If an APQN is assigned to another mediated matrix device, the | |
760 | operation will terminate with an error (EADDRINUSE). | |
761 | ||
762 | In order to successfully assign a domain: | |
763 | ||
764 | * The domain number specified must represent a value from 0 up to the | |
765 | maximum domain number configured for the system. If a domain number | |
766 | higher than the maximum is specified, the operation will terminate with | |
767 | an error (ENODEV). | |
768 | ||
769 | * All APQNs that can be derived from the domain ID and the IDs of | |
770 | the previously assigned adapters must be bound to the vfio_ap device | |
771 | driver. If no domains have yet been assigned, then there must be at least | |
772 | one APQN with the specified APQI bound to the vfio_ap driver. If no such | |
773 | APQNs are bound to the driver, the operation will terminate with an | |
774 | error (EADDRNOTAVAIL). | |
775 | ||
776 | No APQN that can be derived from the domain ID and the IDs of the | |
777 | previously assigned adapters can be assigned to another mediated matrix | |
778 | device. If an APQN is assigned to another mediated matrix device, the | |
779 | operation will terminate with an error (EADDRINUSE). | |
780 | ||
781 | In order to successfully assign a control domain, the domain number | |
782 | specified must represent a value from 0 up to the maximum domain number | |
783 | configured for the system. If a control domain number higher than the maximum | |
784 | is specified, the operation will terminate with an error (ENODEV). | |
785 | ||
786 | 5. Start Guest1: | |
787 | ||
788 | /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ | |
789 | -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ... | |
790 | ||
791 | 7. Start Guest2: | |
792 | ||
793 | /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ | |
794 | -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ... | |
795 | ||
796 | 7. Start Guest3: | |
797 | ||
798 | /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on \ | |
799 | -device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ... | |
800 | ||
801 | When the guest is shut down, the mediated matrix devices may be removed. | |
802 | ||
803 | Using our example again, to remove the mediated matrix device $uuid1: | |
804 | ||
805 | /sys/devices/vfio_ap/matrix/ | |
806 | --- [mdev_supported_types] | |
807 | ------ [vfio_ap-passthrough] | |
808 | --------- [devices] | |
809 | ------------ [$uuid1] | |
810 | --------------- remove | |
811 | ||
812 | ||
813 | echo 1 > remove | |
814 | ||
815 | This will remove all of the mdev matrix device's sysfs structures including | |
816 | the mdev device itself. To recreate and reconfigure the mdev matrix device, | |
817 | all of the steps starting with step 3 will have to be performed again. Note | |
818 | that the remove will fail if a guest using the mdev is still running. | |
819 | ||
820 | It is not necessary to remove an mdev matrix device, but one may want to | |
821 | remove it if no guest will use it during the remaining lifetime of the linux | |
822 | host. If the mdev matrix device is removed, one may want to also reconfigure | |
823 | the pool of adapters and queues reserved for use by the default drivers. | |
824 | ||
825 | Limitations | |
826 | =========== | |
827 | * The KVM/kernel interfaces do not provide a way to prevent restoring an APQN | |
828 | to the default drivers pool of a queue that is still assigned to a mediated | |
829 | device in use by a guest. It is incumbent upon the administrator to | |
830 | ensure there is no mediated device in use by a guest to which the APQN is | |
831 | assigned lest the host be given access to the private data of the AP queue | |
832 | device such as a private key configured specifically for the guest. | |
833 | ||
834 | * Dynamically modifying the AP matrix for a running guest (which would amount to | |
835 | hot(un)plug of AP devices for the guest) is currently not supported | |
836 | ||
837 | * Live guest migration is not supported for guests using AP devices. |