Commit | Line | Data |
---|---|---|
ea81896d MW |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
3 | ====================== | |
4 | USB4 and Thunderbolt | |
5 | ====================== | |
6 | USB4 is the public specification based on Thunderbolt 3 protocol with | |
7 | some differences at the register level among other things. Connection | |
8 | manager is an entity running on the host router (host controller) | |
9 | responsible for enumerating routers and establishing tunnels. A | |
10 | connection manager can be implemented either in firmware or software. | |
11 | Typically PCs come with a firmware connection manager for Thunderbolt 3 | |
12 | and early USB4 capable systems. Apple systems on the other hand use | |
13 | software connection manager and the later USB4 compliant devices follow | |
14 | the suit. | |
15 | ||
16 | The Linux Thunderbolt driver supports both and can detect at runtime which | |
17 | connection manager implementation is to be used. To be on the safe side the | |
18 | software connection manager in Linux also advertises security level | |
19 | ``user`` which means PCIe tunneling is disabled by default. The | |
20 | documentation below applies to both implementations with the exception that | |
21 | the software connection manager only supports ``user`` security level and | |
22 | is expected to be accompanied with an IOMMU based DMA protection. | |
23 | ||
24 | Security levels and how to use them | |
25 | ----------------------------------- | |
163f1511 MW |
26 | The interface presented here is not meant for end users. Instead there |
27 | should be a userspace tool that handles all the low-level details, keeps | |
54e36a2d | 28 | a database of the authorized devices and prompts users for new connections. |
163f1511 MW |
29 | |
30 | More details about the sysfs interface for Thunderbolt devices can be | |
31 | found in ``Documentation/ABI/testing/sysfs-bus-thunderbolt``. | |
32 | ||
33 | Those users who just want to connect any device without any sort of | |
54e36a2d | 34 | manual work can add following line to |
163f1511 MW |
35 | ``/etc/udev/rules.d/99-local.rules``:: |
36 | ||
37 | ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" | |
38 | ||
39 | This will authorize all devices automatically when they appear. However, | |
40 | keep in mind that this bypasses the security levels and makes the system | |
41 | vulnerable to DMA attacks. | |
42 | ||
54e36a2d | 43 | Starting with Intel Falcon Ridge Thunderbolt controller there are 4 |
6fc14e1a MW |
44 | security levels available. Intel Titan Ridge added one more security level |
45 | (usbonly). The reason for these is the fact that the connected devices can | |
46 | be DMA masters and thus read contents of the host memory without CPU and OS | |
47 | knowing about it. There are ways to prevent this by setting up an IOMMU but | |
48 | it is not always available for various reasons. | |
163f1511 | 49 | |
3cd542e6 MW |
50 | Some USB4 systems have a BIOS setting to disable PCIe tunneling. This is |
51 | treated as another security level (nopcie). | |
52 | ||
163f1511 MW |
53 | The security levels are as follows: |
54 | ||
55 | none | |
56 | All devices are automatically connected by the firmware. No user | |
57 | approval is needed. In BIOS settings this is typically called | |
58 | *Legacy mode*. | |
59 | ||
60 | user | |
61 | User is asked whether the device is allowed to be connected. | |
62 | Based on the device identification information available through | |
54e36a2d | 63 | ``/sys/bus/thunderbolt/devices``, the user then can make the decision. |
163f1511 MW |
64 | In BIOS settings this is typically called *Unique ID*. |
65 | ||
66 | secure | |
67 | User is asked whether the device is allowed to be connected. In | |
68 | addition to UUID the device (if it supports secure connect) is sent | |
69 | a challenge that should match the expected one based on a random key | |
54e36a2d | 70 | written to the ``key`` sysfs attribute. In BIOS settings this is |
163f1511 MW |
71 | typically called *One time saved key*. |
72 | ||
73 | dponly | |
74 | The firmware automatically creates tunnels for Display Port and | |
75 | USB. No PCIe tunneling is done. In BIOS settings this is | |
76 | typically called *Display Port Only*. | |
77 | ||
6fc14e1a MW |
78 | usbonly |
79 | The firmware automatically creates tunnels for the USB controller and | |
80 | Display Port in a dock. All PCIe links downstream of the dock are | |
81 | removed. | |
82 | ||
3cd542e6 MW |
83 | nopcie |
84 | PCIe tunneling is disabled/forbidden from the BIOS. Available in some | |
85 | USB4 systems. | |
86 | ||
163f1511 MW |
87 | The current security level can be read from |
88 | ``/sys/bus/thunderbolt/devices/domainX/security`` where ``domainX`` is | |
89 | the Thunderbolt domain the host controller manages. There is typically | |
90 | one domain per Thunderbolt host controller. | |
91 | ||
92 | If the security level reads as ``user`` or ``secure`` the connected | |
93 | device must be authorized by the user before PCIe tunnels are created | |
94 | (e.g the PCIe device appears). | |
95 | ||
96 | Each Thunderbolt device plugged in will appear in sysfs under | |
97 | ``/sys/bus/thunderbolt/devices``. The device directory carries | |
98 | information that can be used to identify the particular device, | |
99 | including its name and UUID. | |
100 | ||
101 | Authorizing devices when security level is ``user`` or ``secure`` | |
102 | ----------------------------------------------------------------- | |
103 | When a device is plugged in it will appear in sysfs as follows:: | |
104 | ||
105 | /sys/bus/thunderbolt/devices/0-1/authorized - 0 | |
106 | /sys/bus/thunderbolt/devices/0-1/device - 0x8004 | |
107 | /sys/bus/thunderbolt/devices/0-1/device_name - Thunderbolt to FireWire Adapter | |
108 | /sys/bus/thunderbolt/devices/0-1/vendor - 0x1 | |
109 | /sys/bus/thunderbolt/devices/0-1/vendor_name - Apple, Inc. | |
110 | /sys/bus/thunderbolt/devices/0-1/unique_id - e0376f00-0300-0100-ffff-ffffffffffff | |
111 | ||
112 | The ``authorized`` attribute reads 0 which means no PCIe tunnels are | |
54e36a2d | 113 | created yet. The user can authorize the device by simply entering:: |
163f1511 MW |
114 | |
115 | # echo 1 > /sys/bus/thunderbolt/devices/0-1/authorized | |
116 | ||
117 | This will create the PCIe tunnels and the device is now connected. | |
118 | ||
119 | If the device supports secure connect, and the domain security level is | |
120 | set to ``secure``, it has an additional attribute ``key`` which can hold | |
54e36a2d | 121 | a random 32-byte value used for authorization and challenging the device in |
163f1511 MW |
122 | future connects:: |
123 | ||
124 | /sys/bus/thunderbolt/devices/0-3/authorized - 0 | |
125 | /sys/bus/thunderbolt/devices/0-3/device - 0x305 | |
126 | /sys/bus/thunderbolt/devices/0-3/device_name - AKiTiO Thunder3 PCIe Box | |
127 | /sys/bus/thunderbolt/devices/0-3/key - | |
128 | /sys/bus/thunderbolt/devices/0-3/vendor - 0x41 | |
129 | /sys/bus/thunderbolt/devices/0-3/vendor_name - inXtron | |
130 | /sys/bus/thunderbolt/devices/0-3/unique_id - dc010000-0000-8508-a22d-32ca6421cb16 | |
131 | ||
132 | Notice the key is empty by default. | |
133 | ||
54e36a2d | 134 | If the user does not want to use secure connect they can just ``echo 1`` |
163f1511 | 135 | to the ``authorized`` attribute and the PCIe tunnels will be created in |
54e36a2d | 136 | the same way as in the ``user`` security level. |
163f1511 MW |
137 | |
138 | If the user wants to use secure connect, the first time the device is | |
54e36a2d | 139 | plugged a key needs to be created and sent to the device:: |
163f1511 MW |
140 | |
141 | # key=$(openssl rand -hex 32) | |
142 | # echo $key > /sys/bus/thunderbolt/devices/0-3/key | |
143 | # echo 1 > /sys/bus/thunderbolt/devices/0-3/authorized | |
144 | ||
145 | Now the device is connected (PCIe tunnels are created) and in addition | |
146 | the key is stored on the device NVM. | |
147 | ||
148 | Next time the device is plugged in the user can verify (challenge) the | |
149 | device using the same key:: | |
150 | ||
151 | # echo $key > /sys/bus/thunderbolt/devices/0-3/key | |
152 | # echo 2 > /sys/bus/thunderbolt/devices/0-3/authorized | |
153 | ||
154 | If the challenge the device returns back matches the one we expect based | |
155 | on the key, the device is connected and the PCIe tunnels are created. | |
54e36a2d | 156 | However, if the challenge fails no tunnels are created and error is |
163f1511 MW |
157 | returned to the user. |
158 | ||
54e36a2d RD |
159 | If the user still wants to connect the device they can either approve |
160 | the device without a key or write a new key and write 1 to the | |
163f1511 MW |
161 | ``authorized`` file to get the new key stored on the device NVM. |
162 | ||
3da88be2 MW |
163 | De-authorizing devices |
164 | ---------------------- | |
165 | It is possible to de-authorize devices by writing ``0`` to their | |
166 | ``authorized`` attribute. This requires support from the connection | |
167 | manager implementation and can be checked by reading domain | |
168 | ``deauthorization`` attribute. If it reads ``1`` then the feature is | |
169 | supported. | |
170 | ||
171 | When a device is de-authorized the PCIe tunnel from the parent device | |
172 | PCIe downstream (or root) port to the device PCIe upstream port is torn | |
173 | down. This is essentially the same thing as PCIe hot-remove and the PCIe | |
174 | toplogy in question will not be accessible anymore until the device is | |
175 | authorized again. If there is storage such as NVMe or similar involved, | |
176 | there is a risk for data loss if the filesystem on that storage is not | |
177 | properly shut down. You have been warned! | |
178 | ||
dcc3c9e3 MW |
179 | DMA protection utilizing IOMMU |
180 | ------------------------------ | |
181 | Recent systems from 2018 and forward with Thunderbolt ports may natively | |
182 | support IOMMU. This means that Thunderbolt security is handled by an IOMMU | |
183 | so connected devices cannot access memory regions outside of what is | |
184 | allocated for them by drivers. When Linux is running on such system it | |
185 | automatically enables IOMMU if not enabled by the user already. These | |
186 | systems can be identified by reading ``1`` from | |
187 | ``/sys/bus/thunderbolt/devices/domainX/iommu_dma_protection`` attribute. | |
188 | ||
189 | The driver does not do anything special in this case but because DMA | |
190 | protection is handled by the IOMMU, security levels (if set) are | |
191 | redundant. For this reason some systems ship with security level set to | |
192 | ``none``. Other systems have security level set to ``user`` in order to | |
193 | support downgrade to older OS, so users who want to automatically | |
194 | authorize devices when IOMMU DMA protection is enabled can use the | |
195 | following ``udev`` rule:: | |
196 | ||
197 | ACTION=="add", SUBSYSTEM=="thunderbolt", ATTRS{iommu_dma_protection}=="1", ATTR{authorized}=="0", ATTR{authorized}="1" | |
198 | ||
dacb1287 KK |
199 | Upgrading NVM on Thunderbolt device, host or retimer |
200 | ---------------------------------------------------- | |
54e36a2d | 201 | Since most of the functionality is handled in firmware running on a |
163f1511 MW |
202 | host controller or a device, it is important that the firmware can be |
203 | upgraded to the latest where possible bugs in it have been fixed. | |
204 | Typically OEMs provide this firmware from their support site. | |
205 | ||
54e36a2d | 206 | There is also a central site which has links where to download firmware |
163f1511 MW |
207 | for some machines: |
208 | ||
209 | `Thunderbolt Updates <https://thunderbolttechnology.net/updates>`_ | |
210 | ||
dacb1287 KK |
211 | Before you upgrade firmware on a device, host or retimer, please make |
212 | sure it is a suitable upgrade. Failing to do that may render the device | |
213 | in a state where it cannot be used properly anymore without special | |
214 | tools! | |
163f1511 MW |
215 | |
216 | Host NVM upgrade on Apple Macs is not supported. | |
217 | ||
218 | Once the NVM image has been downloaded, you need to plug in a | |
219 | Thunderbolt device so that the host controller appears. It does not | |
220 | matter which device is connected (unless you are upgrading NVM on a | |
221 | device - then you need to connect that particular device). | |
222 | ||
54e36a2d | 223 | Note an OEM-specific method to power the controller up ("force power") may |
163f1511 MW |
224 | be available for your system in which case there is no need to plug in a |
225 | Thunderbolt device. | |
226 | ||
227 | After that we can write the firmware to the non-active parts of the NVM | |
228 | of the host or device. As an example here is how Intel NUC6i7KYK (Skull | |
229 | Canyon) Thunderbolt controller NVM is upgraded:: | |
230 | ||
231 | # dd if=KYK_TBT_FW_0018.bin of=/sys/bus/thunderbolt/devices/0-0/nvm_non_active0/nvmem | |
232 | ||
233 | Once the operation completes we can trigger NVM authentication and | |
234 | upgrade process as follows:: | |
235 | ||
236 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/nvm_authenticate | |
237 | ||
238 | If no errors are returned, the host controller shortly disappears. Once | |
239 | it comes back the driver notices it and initiates a full power cycle. | |
240 | After a while the host controller appears again and this time it should | |
241 | be fully functional. | |
242 | ||
54e36a2d | 243 | We can verify that the new NVM firmware is active by running the following |
163f1511 MW |
244 | commands:: |
245 | ||
246 | # cat /sys/bus/thunderbolt/devices/0-0/nvm_authenticate | |
247 | 0x0 | |
248 | # cat /sys/bus/thunderbolt/devices/0-0/nvm_version | |
249 | 18.0 | |
250 | ||
54e36a2d | 251 | If ``nvm_authenticate`` contains anything other than 0x0 it is the error |
163f1511 MW |
252 | code from the last authentication cycle, which means the authentication |
253 | of the NVM image failed. | |
254 | ||
255 | Note names of the NVMem devices ``nvm_activeN`` and ``nvm_non_activeN`` | |
54e36a2d | 256 | depend on the order they are registered in the NVMem subsystem. N in |
163f1511 MW |
257 | the name is the identifier added by the NVMem subsystem. |
258 | ||
3fb10ea4 RM |
259 | Upgrading on-board retimer NVM when there is no cable connected |
260 | --------------------------------------------------------------- | |
261 | If the platform supports, it may be possible to upgrade the retimer NVM | |
262 | firmware even when there is nothing connected to the USB4 | |
263 | ports. When this is the case the ``usb4_portX`` devices have two special | |
264 | attributes: ``offline`` and ``rescan``. The way to upgrade the firmware | |
265 | is to first put the USB4 port into offline mode:: | |
266 | ||
267 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline | |
268 | ||
269 | This step makes sure the port does not respond to any hotplug events, | |
270 | and also ensures the retimers are powered on. The next step is to scan | |
271 | for the retimers:: | |
272 | ||
273 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan | |
274 | ||
275 | This enumerates and adds the on-board retimers. Now retimer NVM can be | |
276 | upgraded in the same way than with cable connected (see previous | |
277 | section). However, the retimer is not disconnected as we are offline | |
278 | mode) so after writing ``1`` to ``nvm_authenticate`` one should wait for | |
279 | 5 or more seconds before running rescan again:: | |
280 | ||
281 | # echo 1 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/rescan | |
282 | ||
283 | This point if everything went fine, the port can be put back to | |
284 | functional state again:: | |
285 | ||
286 | # echo 0 > /sys/bus/thunderbolt/devices/0-0/usb4_port1/offline | |
287 | ||
163f1511 MW |
288 | Upgrading NVM when host controller is in safe mode |
289 | -------------------------------------------------- | |
290 | If the existing NVM is not properly authenticated (or is missing) the | |
54e36a2d RD |
291 | host controller goes into safe mode which means that the only available |
292 | functionality is flashing a new NVM image. When in this mode, reading | |
163f1511 MW |
293 | ``nvm_version`` fails with ``ENODATA`` and the device identification |
294 | information is missing. | |
295 | ||
296 | To recover from this mode, one needs to flash a valid NVM image to the | |
54e36a2d | 297 | host controller in the same way it is done in the previous chapter. |
e69b6c02 AL |
298 | |
299 | Networking over Thunderbolt cable | |
300 | --------------------------------- | |
54e36a2d | 301 | Thunderbolt technology allows software communication between two hosts |
e69b6c02 AL |
302 | connected by a Thunderbolt cable. |
303 | ||
54e36a2d | 304 | It is possible to tunnel any kind of traffic over a Thunderbolt link but |
e69b6c02 AL |
305 | currently we only support Apple ThunderboltIP protocol. |
306 | ||
54e36a2d RD |
307 | If the other host is running Windows or macOS, the only thing you need to |
308 | do is to connect a Thunderbolt cable between the two hosts; the | |
309 | ``thunderbolt-net`` driver is loaded automatically. If the other host is | |
310 | also Linux you should load ``thunderbolt-net`` manually on one host (it | |
311 | does not matter which one):: | |
e69b6c02 AL |
312 | |
313 | # modprobe thunderbolt-net | |
314 | ||
315 | This triggers module load on the other host automatically. If the driver | |
316 | is built-in to the kernel image, there is no need to do anything. | |
317 | ||
318 | The driver will create one virtual ethernet interface per Thunderbolt | |
319 | port which are named like ``thunderbolt0`` and so on. From this point | |
320 | you can either use standard userspace tools like ``ifconfig`` to | |
54e36a2d | 321 | configure the interface or let your GUI handle it automatically. |
07c455ee | 322 | |
ce6a9002 ML |
323 | Forcing power |
324 | ------------- | |
325 | Many OEMs include a method that can be used to force the power of a | |
54e36a2d | 326 | Thunderbolt controller to an "On" state even if nothing is connected. |
ce6a9002 ML |
327 | If supported by your machine this will be exposed by the WMI bus with |
328 | a sysfs attribute called "force_power". | |
329 | ||
330 | For example the intel-wmi-thunderbolt driver exposes this attribute in: | |
cce1fea5 | 331 | /sys/bus/wmi/devices/86CCFD48-205E-4A77-9C48-2021CBEDE341/force_power |
ce6a9002 ML |
332 | |
333 | To force the power to on, write 1 to this attribute file. | |
334 | To disable force power, write 0 to this attribute file. | |
335 | ||
336 | Note: it's currently not possible to query the force power state of a platform. |