Commit | Line | Data |
---|---|---|
3b9bae02 CD |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | .. include:: <isonum.txt> | |
1da177e4 | 3 | |
3b9bae02 CD |
4 | ========================== |
5 | The MSI Driver Guide HOWTO | |
6 | ========================== | |
7 | ||
8 | :Authors: Tom L Nguyen; Martine Silbermann; Matthew Wilcox | |
9 | ||
10 | :Copyright: 2003, 2008 Intel Corporation | |
11 | ||
12 | About this guide | |
13 | ================ | |
1da177e4 | 14 | |
c41ade2e MW |
15 | This guide describes the basics of Message Signaled Interrupts (MSIs), |
16 | the advantages of using MSI over traditional interrupt mechanisms, how | |
17 | to change your driver to use MSI or MSI-X and some basic diagnostics to | |
18 | try if a device doesn't support MSIs. | |
19 | ||
20 | ||
3b9bae02 CD |
21 | What are MSIs? |
22 | ============== | |
c41ade2e MW |
23 | |
24 | A Message Signaled Interrupt is a write from the device to a special | |
25 | address which causes an interrupt to be received by the CPU. | |
26 | ||
27 | The MSI capability was first specified in PCI 2.2 and was later enhanced | |
28 | in PCI 3.0 to allow each interrupt to be masked individually. The MSI-X | |
29 | capability was also introduced with PCI 3.0. It supports more interrupts | |
30 | per device than MSI and allows interrupts to be independently configured. | |
31 | ||
32 | Devices may support both MSI and MSI-X, but only one can be enabled at | |
33 | a time. | |
34 | ||
35 | ||
3b9bae02 CD |
36 | Why use MSIs? |
37 | ============= | |
c41ade2e MW |
38 | |
39 | There are three reasons why using MSIs can give an advantage over | |
40 | traditional pin-based interrupts. | |
41 | ||
42 | Pin-based PCI interrupts are often shared amongst several devices. | |
43 | To support this, the kernel must call each interrupt handler associated | |
44 | with an interrupt, which leads to reduced performance for the system as | |
45 | a whole. MSIs are never shared, so this problem cannot arise. | |
46 | ||
47 | When a device writes data to memory, then raises a pin-based interrupt, | |
48 | it is possible that the interrupt may arrive before all the data has | |
49 | arrived in memory (this becomes more likely with devices behind PCI-PCI | |
50 | bridges). In order to ensure that all the data has arrived in memory, | |
51 | the interrupt handler must read a register on the device which raised | |
52 | the interrupt. PCI transaction ordering rules require that all the data | |
891f6925 | 53 | arrive in memory before the value may be returned from the register. |
c41ade2e MW |
54 | Using MSIs avoids this problem as the interrupt-generating write cannot |
55 | pass the data writes, so by the time the interrupt is raised, the driver | |
56 | knows that all the data has arrived in memory. | |
57 | ||
58 | PCI devices can only support a single pin-based interrupt per function. | |
59 | Often drivers have to query the device to find out what event has | |
60 | occurred, slowing down interrupt handling for the common case. With | |
61 | MSIs, a device can support more interrupts, allowing each interrupt | |
62 | to be specialised to a different purpose. One possible design gives | |
63 | infrequent conditions (such as errors) their own interrupt which allows | |
64 | the driver to handle the normal interrupt handling path more efficiently. | |
65 | Other possible designs include giving one interrupt to each packet queue | |
66 | in a network card or each port in a storage controller. | |
67 | ||
68 | ||
3b9bae02 CD |
69 | How to use MSIs |
70 | =============== | |
c41ade2e MW |
71 | |
72 | PCI devices are initialised to use pin-based interrupts. The device | |
73 | driver has to set up the device to use MSI or MSI-X. Not all machines | |
74 | support MSIs correctly, and for those machines, the APIs described below | |
75 | will simply fail and the device will continue to use pin-based interrupts. | |
76 | ||
3b9bae02 CD |
77 | Include kernel support for MSIs |
78 | ------------------------------- | |
c41ade2e MW |
79 | |
80 | To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI | |
81 | option enabled. This option is only available on some architectures, | |
82 | and it may depend on some other options also being set. For example, | |
83 | on x86, you must also enable X86_UP_APIC or SMP in order to see the | |
84 | CONFIG_PCI_MSI option. | |
85 | ||
3b9bae02 CD |
86 | Using MSI |
87 | --------- | |
c41ade2e | 88 | |
aff17164 CH |
89 | Most of the hard work is done for the driver in the PCI layer. The driver |
90 | simply has to request that the PCI layer set up the MSI capability for this | |
c41ade2e MW |
91 | device. |
92 | ||
aff17164 | 93 | To automatically use MSI or MSI-X interrupt vectors, use the following |
3b9bae02 | 94 | function:: |
7918b2dc | 95 | |
aff17164 CH |
96 | int pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs, |
97 | unsigned int max_vecs, unsigned int flags); | |
7918b2dc | 98 | |
aff17164 CH |
99 | which allocates up to max_vecs interrupt vectors for a PCI device. It |
100 | returns the number of vectors allocated or a negative error. If the device | |
101 | has a requirements for a minimum number of vectors the driver can pass a | |
102 | min_vecs argument set to this limit, and the PCI core will return -ENOSPC | |
103 | if it can't meet the minimum number of vectors. | |
7918b2dc | 104 | |
4fe0d154 | 105 | The flags argument is used to specify which type of interrupt can be used |
b21f89b0 | 106 | by the device and the driver (PCI_IRQ_INTX, PCI_IRQ_MSI, PCI_IRQ_MSIX). |
4fe0d154 CH |
107 | A convenient short-hand (PCI_IRQ_ALL_TYPES) is also available to ask for |
108 | any possible kind of interrupt. If the PCI_IRQ_AFFINITY flag is set, | |
109 | pci_alloc_irq_vectors() will spread the interrupts around the available CPUs. | |
4ef33685 | 110 | |
aff17164 | 111 | To get the Linux IRQ numbers passed to request_irq() and free_irq() and the |
3b9bae02 | 112 | vectors, use the following function:: |
1da177e4 | 113 | |
aff17164 | 114 | int pci_irq_vector(struct pci_dev *dev, unsigned int nr); |
1da177e4 | 115 | |
aff17164 | 116 | Any allocated resources should be freed before removing the device using |
3b9bae02 | 117 | the following function:: |
1c8d7b0a | 118 | |
aff17164 | 119 | void pci_free_irq_vectors(struct pci_dev *dev); |
1c8d7b0a | 120 | |
aff17164 CH |
121 | If a device supports both MSI-X and MSI capabilities, this API will use the |
122 | MSI-X facilities in preference to the MSI facilities. MSI-X supports any | |
123 | number of interrupts between 1 and 2048. In contrast, MSI is restricted to | |
124 | a maximum of 32 interrupts (and must be a power of two). In addition, the | |
125 | MSI interrupt vectors must be allocated consecutively, so the system might | |
126 | not be able to allocate as many vectors for MSI as it could for MSI-X. On | |
127 | some platforms, MSI interrupts must all be targeted at the same set of CPUs | |
128 | whereas MSI-X interrupts can all be targeted at different CPUs. | |
1c8d7b0a | 129 | |
aff17164 CH |
130 | If a device supports neither MSI-X or MSI it will fall back to a single |
131 | legacy IRQ vector. | |
1c8d7b0a | 132 | |
aff17164 CH |
133 | The typical usage of MSI or MSI-X interrupts is to allocate as many vectors |
134 | as possible, likely up to the limit supported by the device. If nvec is | |
135 | larger than the number supported by the device it will automatically be | |
136 | capped to the supported limit, so there is no need to query the number of | |
3b9bae02 | 137 | vectors supported beforehand:: |
302a2523 | 138 | |
4fe0d154 | 139 | nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_ALL_TYPES) |
aff17164 CH |
140 | if (nvec < 0) |
141 | goto out_err; | |
302a2523 AG |
142 | |
143 | If a driver is unable or unwilling to deal with a variable number of MSI | |
aff17164 CH |
144 | interrupts it can request a particular number of interrupts by passing that |
145 | number to pci_alloc_irq_vectors() function as both 'min_vecs' and | |
3b9bae02 | 146 | 'max_vecs' parameters:: |
c41ade2e | 147 | |
4fe0d154 | 148 | ret = pci_alloc_irq_vectors(pdev, nvec, nvec, PCI_IRQ_ALL_TYPES); |
aff17164 CH |
149 | if (ret < 0) |
150 | goto out_err; | |
ff1aa430 | 151 | |
aff17164 CH |
152 | The most notorious example of the request type described above is enabling |
153 | the single MSI mode for a device. It could be done by passing two 1s as | |
3b9bae02 | 154 | 'min_vecs' and 'max_vecs':: |
ff1aa430 | 155 | |
4fe0d154 | 156 | ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES); |
aff17164 CH |
157 | if (ret < 0) |
158 | goto out_err; | |
ff1aa430 | 159 | |
aff17164 | 160 | Some devices might not support using legacy line interrupts, in which case |
3b9bae02 | 161 | the driver can specify that only MSI or MSI-X is acceptable:: |
ff1aa430 | 162 | |
4fe0d154 | 163 | nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_MSI | PCI_IRQ_MSIX); |
aff17164 CH |
164 | if (nvec < 0) |
165 | goto out_err; | |
ff1aa430 | 166 | |
3b9bae02 CD |
167 | Legacy APIs |
168 | ----------- | |
c41ade2e | 169 | |
aff17164 | 170 | The following old APIs to enable and disable MSI or MSI-X interrupts should |
3b9bae02 | 171 | not be used in new code:: |
c41ade2e | 172 | |
aff17164 | 173 | pci_enable_msi() /* deprecated */ |
aff17164 CH |
174 | pci_disable_msi() /* deprecated */ |
175 | pci_enable_msix_range() /* deprecated */ | |
176 | pci_enable_msix_exact() /* deprecated */ | |
177 | pci_disable_msix() /* deprecated */ | |
c41ade2e | 178 | |
aff17164 CH |
179 | Additionally there are APIs to provide the number of supported MSI or MSI-X |
180 | vectors: pci_msi_vec_count() and pci_msix_vec_count(). In general these | |
181 | should be avoided in favor of letting pci_alloc_irq_vectors() cap the | |
182 | number of vectors. If you have a legitimate special use case for the count | |
183 | of vectors we might have to revisit that decision and add a | |
184 | pci_nr_irq_vectors() helper that handles MSI and MSI-X transparently. | |
c41ade2e | 185 | |
3b9bae02 CD |
186 | Considerations when using MSIs |
187 | ------------------------------ | |
c41ade2e | 188 | |
3b9bae02 CD |
189 | Spinlocks |
190 | ~~~~~~~~~ | |
c41ade2e MW |
191 | |
192 | Most device drivers have a per-device spinlock which is taken in the | |
193 | interrupt handler. With pin-based interrupts or a single MSI, it is not | |
194 | necessary to disable interrupts (Linux guarantees the same interrupt will | |
195 | not be re-entered). If a device uses multiple interrupts, the driver | |
196 | must disable interrupts while the lock is held. If the device sends | |
197 | a different interrupt, the driver will deadlock trying to recursively | |
2f9d7389 VR |
198 | acquire the spinlock. Such deadlocks can be avoided by using |
199 | spin_lock_irqsave() or spin_lock_irq() which disable local interrupts | |
ff41c419 | 200 | and acquire the lock (see Documentation/kernel-hacking/locking.rst). |
c41ade2e | 201 | |
3b9bae02 CD |
202 | How to tell whether MSI/MSI-X is enabled on a device |
203 | ---------------------------------------------------- | |
c41ade2e MW |
204 | |
205 | Using 'lspci -v' (as root) may show some devices with "MSI", "Message | |
206 | Signalled Interrupts" or "MSI-X" capabilities. Each of these capabilities | |
4979de6e | 207 | has an 'Enable' flag which is followed with either "+" (enabled) |
c41ade2e MW |
208 | or "-" (disabled). |
209 | ||
210 | ||
3b9bae02 CD |
211 | MSI quirks |
212 | ========== | |
c41ade2e MW |
213 | |
214 | Several PCI chipsets or devices are known not to support MSIs. | |
215 | The PCI stack provides three ways to disable MSIs: | |
216 | ||
217 | 1. globally | |
218 | 2. on all devices behind a specific bridge | |
219 | 3. on a single device | |
220 | ||
3b9bae02 CD |
221 | Disabling MSIs globally |
222 | ----------------------- | |
c41ade2e MW |
223 | |
224 | Some host chipsets simply don't support MSIs properly. If we're | |
225 | lucky, the manufacturer knows this and has indicated it in the ACPI | |
4979de6e | 226 | FADT table. In this case, Linux automatically disables MSIs. |
c41ade2e MW |
227 | Some boards don't include this information in the table and so we have |
228 | to detect them ourselves. The complete list of these is found near the | |
229 | quirk_disable_all_msi() function in drivers/pci/quirks.c. | |
230 | ||
231 | If you have a board which has problems with MSIs, you can pass pci=nomsi | |
232 | on the kernel command line to disable MSIs on all devices. It would be | |
233 | in your best interests to report the problem to linux-pci@vger.kernel.org | |
234 | including a full 'lspci -v' so we can add the quirks to the kernel. | |
235 | ||
3b9bae02 CD |
236 | Disabling MSIs below a bridge |
237 | ----------------------------- | |
c41ade2e | 238 | |
9ca65c37 | 239 | Some PCI bridges are not able to route MSIs between buses properly. |
c41ade2e MW |
240 | In this case, MSIs must be disabled on all devices behind the bridge. |
241 | ||
242 | Some bridges allow you to enable MSIs by changing some bits in their | |
243 | PCI configuration space (especially the Hypertransport chipsets such | |
244 | as the nVidia nForce and Serverworks HT2000). As with host chipsets, | |
245 | Linux mostly knows about them and automatically enables MSIs if it can. | |
e6b85a1f | 246 | If you have a bridge unknown to Linux, you can enable |
c41ade2e | 247 | MSIs in configuration space using whatever method you know works, then |
3b9bae02 | 248 | enable MSIs on that bridge by doing:: |
c41ade2e MW |
249 | |
250 | echo 1 > /sys/bus/pci/devices/$bridge/msi_bus | |
251 | ||
252 | where $bridge is the PCI address of the bridge you've enabled (eg | |
253 | 0000:00:0e.0). | |
254 | ||
255 | To disable MSIs, echo 0 instead of 1. Changing this value should be | |
1b8386f6 | 256 | done with caution as it could break interrupt handling for all devices |
c41ade2e MW |
257 | below this bridge. |
258 | ||
259 | Again, please notify linux-pci@vger.kernel.org of any bridges that need | |
260 | special handling. | |
261 | ||
3b9bae02 CD |
262 | Disabling MSIs on a single device |
263 | --------------------------------- | |
c41ade2e MW |
264 | |
265 | Some devices are known to have faulty MSI implementations. Usually this | |
c2b65e18 | 266 | is handled in the individual device driver, but occasionally it's necessary |
c41ade2e MW |
267 | to handle this with a quirk. Some drivers have an option to disable use |
268 | of MSI. While this is a convenient workaround for the driver author, | |
305af08c | 269 | it is not good practice, and should not be emulated. |
c41ade2e | 270 | |
3b9bae02 CD |
271 | Finding why MSIs are disabled on a device |
272 | ----------------------------------------- | |
c41ade2e MW |
273 | |
274 | From the above three sections, you can see that there are many reasons | |
275 | why MSIs may not be enabled for a given device. Your first step should | |
276 | be to examine your dmesg carefully to determine whether MSIs are enabled | |
277 | for your machine. You should also check your .config to be sure you | |
278 | have enabled CONFIG_PCI_MSI. | |
279 | ||
3b9bae02 CD |
280 | Then, 'lspci -t' gives the list of bridges above a device. Reading |
281 | `/sys/bus/pci/devices/*/msi_bus` will tell you whether MSIs are enabled (1) | |
c41ade2e MW |
282 | or disabled (0). If 0 is found in any of the msi_bus files belonging |
283 | to bridges between the PCI root and the device, MSIs are disabled. | |
284 | ||
285 | It is also worth checking the device driver to see whether it supports MSIs. | |
7730c3be | 286 | For example, it may contain calls to pci_alloc_irq_vectors() with the |
4fe03955 | 287 | PCI_IRQ_MSI or PCI_IRQ_MSIX flags. |
88614075 AD |
288 | |
289 | ||
290 | List of device drivers MSI(-X) APIs | |
291 | =================================== | |
292 | ||
b58d6d89 | 293 | The PCI/MSI subsystem has a dedicated C file for its exported device driver |
88614075 AD |
294 | APIs — `drivers/pci/msi/api.c`. The following functions are exported: |
295 | ||
296 | .. kernel-doc:: drivers/pci/msi/api.c | |
297 | :export: |