Commit | Line | Data |
---|---|---|
00b96c0e FB |
1 | ======================================================== |
2 | OpenCAPI (Open Coherent Accelerator Processor Interface) | |
3 | ======================================================== | |
4 | ||
5 | OpenCAPI is an interface between processors and accelerators. It aims | |
6 | at being low-latency and high-bandwidth. The specification is | |
7 | developed by the `OpenCAPI Consortium <http://opencapi.org/>`_. | |
8 | ||
9 | It allows an accelerator (which could be a FPGA, ASICs, ...) to access | |
10 | the host memory coherently, using virtual addresses. An OpenCAPI | |
11 | device can also host its own memory, that can be accessed from the | |
12 | host. | |
13 | ||
14 | OpenCAPI is known in linux as 'ocxl', as the open, processor-agnostic | |
15 | evolution of 'cxl' (the driver for the IBM CAPI interface for | |
16 | powerpc), which was named that way to avoid confusion with the ISDN | |
17 | CAPI subsystem. | |
18 | ||
19 | ||
20 | High-level view | |
21 | =============== | |
22 | ||
23 | OpenCAPI defines a Data Link Layer (DL) and Transaction Layer (TL), to | |
24 | be implemented on top of a physical link. Any processor or device | |
25 | implementing the DL and TL can start sharing memory. | |
26 | ||
27 | :: | |
28 | ||
29 | +-----------+ +-------------+ | |
30 | | | | | | |
31 | | | | Accelerated | | |
32 | | Processor | | Function | | |
33 | | | +--------+ | Unit | +--------+ | |
34 | | |--| Memory | | (AFU) |--| Memory | | |
35 | | | +--------+ | | +--------+ | |
36 | +-----------+ +-------------+ | |
37 | | | | |
38 | +-----------+ +-------------+ | |
39 | | TL | | TLX | | |
40 | +-----------+ +-------------+ | |
41 | | | | |
42 | +-----------+ +-------------+ | |
43 | | DL | | DLX | | |
44 | +-----------+ +-------------+ | |
45 | | | | |
46 | | PHY | | |
47 | +---------------------------------------+ | |
48 | ||
49 | ||
50 | ||
51 | Device discovery | |
52 | ================ | |
53 | ||
54 | OpenCAPI relies on a PCI-like configuration space, implemented on the | |
55 | device. So the host can discover AFUs by querying the config space. | |
56 | ||
57 | OpenCAPI devices in Linux are treated like PCI devices (with a few | |
58 | caveats). The firmware is expected to abstract the hardware as if it | |
59 | was a PCI link. A lot of the existing PCI infrastructure is reused: | |
60 | devices are scanned and BARs are assigned during the standard PCI | |
61 | enumeration. Commands like 'lspci' can therefore be used to see what | |
62 | devices are available. | |
63 | ||
64 | The configuration space defines the AFU(s) that can be found on the | |
65 | physical adapter, such as its name, how many memory contexts it can | |
66 | work with, the size of its MMIO areas, ... | |
67 | ||
68 | ||
69 | ||
70 | MMIO | |
71 | ==== | |
72 | ||
73 | OpenCAPI defines two MMIO areas for each AFU: | |
74 | ||
75 | * the global MMIO area, with registers pertinent to the whole AFU. | |
76 | * a per-process MMIO area, which has a fixed size for each context. | |
77 | ||
78 | ||
79 | ||
80 | AFU interrupts | |
81 | ============== | |
82 | ||
83 | OpenCAPI includes the possibility for an AFU to send an interrupt to a | |
84 | host process. It is done through a 'intrp_req' defined in the | |
85 | Transaction Layer, specifying a 64-bit object handle which defines the | |
86 | interrupt. | |
87 | ||
88 | The driver allows a process to allocate an interrupt and obtain its | |
89 | 64-bit object handle, that can be passed to the AFU. | |
90 | ||
91 | ||
92 | ||
93 | char devices | |
94 | ============ | |
95 | ||
96 | The driver creates one char device per AFU found on the physical | |
97 | device. A physical device may have multiple functions and each | |
98 | function can have multiple AFUs. At the time of this writing though, | |
99 | it has only been tested with devices exporting only one AFU. | |
100 | ||
101 | Char devices can be found in /dev/ocxl/ and are named as: | |
102 | /dev/ocxl/<AFU name>.<location>.<index> | |
103 | ||
104 | where <AFU name> is a max 20-character long name, as found in the | |
105 | config space of the AFU. | |
106 | <location> is added by the driver and can help distinguish devices | |
107 | when a system has more than one instance of the same OpenCAPI device. | |
108 | <index> is also to help distinguish AFUs in the unlikely case where a | |
109 | device carries multiple copies of the same AFU. | |
110 | ||
111 | ||
112 | ||
113 | Sysfs class | |
114 | =========== | |
115 | ||
116 | An ocxl class is added for the devices representing the AFUs. See | |
117 | /sys/class/ocxl. The layout is described in | |
118 | Documentation/ABI/testing/sysfs-class-ocxl | |
119 | ||
120 | ||
121 | ||
122 | User API | |
123 | ======== | |
124 | ||
125 | open | |
126 | ---- | |
127 | ||
128 | Based on the AFU definition found in the config space, an AFU may | |
129 | support working with more than one memory context, in which case the | |
130 | associated char device may be opened multiple times by different | |
131 | processes. | |
132 | ||
133 | ||
134 | ioctl | |
135 | ----- | |
136 | ||
137 | OCXL_IOCTL_ATTACH: | |
138 | ||
139 | Attach the memory context of the calling process to the AFU so that | |
140 | the AFU can access its memory. | |
141 | ||
142 | OCXL_IOCTL_IRQ_ALLOC: | |
143 | ||
144 | Allocate an AFU interrupt and return an identifier. | |
145 | ||
146 | OCXL_IOCTL_IRQ_FREE: | |
147 | ||
148 | Free a previously allocated AFU interrupt. | |
149 | ||
150 | OCXL_IOCTL_IRQ_SET_FD: | |
151 | ||
152 | Associate an event fd to an AFU interrupt so that the user process | |
153 | can be notified when the AFU sends an interrupt. | |
154 | ||
e7666d04 AS |
155 | OCXL_IOCTL_GET_METADATA: |
156 | ||
157 | Obtains configuration information from the card, such at the size of | |
158 | MMIO areas, the AFU version, and the PASID for the current context. | |
159 | ||
721c551d AS |
160 | OCXL_IOCTL_ENABLE_P9_WAIT: |
161 | ||
162 | Allows the AFU to wake a userspace thread executing 'wait'. Returns | |
163 | information to userspace to allow it to configure the AFU. Note that | |
164 | this is only available on POWER9. | |
165 | ||
166 | OCXL_IOCTL_GET_FEATURES: | |
167 | ||
168 | Reports on which CPU features that affect OpenCAPI are usable from | |
169 | userspace. | |
170 | ||
00b96c0e FB |
171 | |
172 | mmap | |
173 | ---- | |
174 | ||
175 | A process can mmap the per-process MMIO area for interactions with the | |
176 | AFU. |