Commit | Line | Data |
---|---|---|
b4b8faa1 MK |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
3 | ====== | |
4 | AF_XDP | |
5 | ====== | |
6 | ||
7 | Overview | |
8 | ======== | |
9 | ||
10 | AF_XDP is an address family that is optimized for high performance | |
11 | packet processing. | |
12 | ||
13 | This document assumes that the reader is familiar with BPF and XDP. If | |
14 | not, the Cilium project has an excellent reference guide at | |
bbff2f32 | 15 | http://cilium.readthedocs.io/en/latest/bpf/. |
b4b8faa1 MK |
16 | |
17 | Using the XDP_REDIRECT action from an XDP program, the program can | |
18 | redirect ingress frames to other XDP enabled netdevs, using the | |
19 | bpf_redirect_map() function. AF_XDP sockets enable the possibility for | |
20 | XDP programs to redirect frames to a memory buffer in a user-space | |
21 | application. | |
22 | ||
23 | An AF_XDP socket (XSK) is created with the normal socket() | |
24 | syscall. Associated with each XSK are two rings: the RX ring and the | |
25 | TX ring. A socket can receive packets on the RX ring and it can send | |
26 | packets on the TX ring. These rings are registered and sized with the | |
27 | setsockopts XDP_RX_RING and XDP_TX_RING, respectively. It is mandatory | |
28 | to have at least one of these rings for each socket. An RX or TX | |
29 | descriptor ring points to a data buffer in a memory area called a | |
30 | UMEM. RX and TX can share the same UMEM so that a packet does not have | |
31 | to be copied between RX and TX. Moreover, if a packet needs to be kept | |
32 | for a while due to a possible retransmit, the descriptor that points | |
33 | to that packet can be changed to point to another and reused right | |
34 | away. This again avoids copying data. | |
35 | ||
bbff2f32 BT |
36 | The UMEM consists of a number of equally sized chunks. A descriptor in |
37 | one of the rings references a frame by referencing its addr. The addr | |
38 | is simply an offset within the entire UMEM region. The user space | |
39 | allocates memory for this UMEM using whatever means it feels is most | |
40 | appropriate (malloc, mmap, huge pages, etc). This memory area is then | |
41 | registered with the kernel using the new setsockopt XDP_UMEM_REG. The | |
42 | UMEM also has two rings: the FILL ring and the COMPLETION ring. The | |
e0e4f8e9 | 43 | FILL ring is used by the application to send down addr for the kernel |
bbff2f32 BT |
44 | to fill in with RX packet data. References to these frames will then |
45 | appear in the RX ring once each packet has been received. The | |
e0e4f8e9 | 46 | COMPLETION ring, on the other hand, contains frame addr that the |
bbff2f32 BT |
47 | kernel has transmitted completely and can now be used again by user |
48 | space, for either TX or RX. Thus, the frame addrs appearing in the | |
e0e4f8e9 | 49 | COMPLETION ring are addrs that were previously transmitted using the |
bbff2f32 BT |
50 | TX ring. In summary, the RX and FILL rings are used for the RX path |
51 | and the TX and COMPLETION rings are used for the TX path. | |
b4b8faa1 MK |
52 | |
53 | The socket is then finally bound with a bind() call to a device and a | |
54 | specific queue id on that device, and it is not until bind is | |
55 | completed that traffic starts to flow. | |
56 | ||
57 | The UMEM can be shared between processes, if desired. If a process | |
58 | wants to do this, it simply skips the registration of the UMEM and its | |
59 | corresponding two rings, sets the XDP_SHARED_UMEM flag in the bind | |
60 | call and submits the XSK of the process it would like to share UMEM | |
61 | with as well as its own newly created XSK socket. The new process will | |
bbff2f32 BT |
62 | then receive frame addr references in its own RX ring that point to |
63 | this shared UMEM. Note that since the ring structures are | |
64 | single-consumer / single-producer (for performance reasons), the new | |
65 | process has to create its own socket with associated RX and TX rings, | |
66 | since it cannot share this with the other process. This is also the | |
67 | reason that there is only one set of FILL and COMPLETION rings per | |
68 | UMEM. It is the responsibility of a single process to handle the UMEM. | |
b4b8faa1 MK |
69 | |
70 | How is then packets distributed from an XDP program to the XSKs? There | |
71 | is a BPF map called XSKMAP (or BPF_MAP_TYPE_XSKMAP in full). The | |
72 | user-space application can place an XSK at an arbitrary place in this | |
73 | map. The XDP program can then redirect a packet to a specific index in | |
74 | this map and at this point XDP validates that the XSK in that map was | |
75 | indeed bound to that device and ring number. If not, the packet is | |
76 | dropped. If the map is empty at that index, the packet is also | |
77 | dropped. This also means that it is currently mandatory to have an XDP | |
78 | program loaded (and one XSK in the XSKMAP) to be able to get any | |
79 | traffic to user space through the XSK. | |
80 | ||
81 | AF_XDP can operate in two different modes: XDP_SKB and XDP_DRV. If the | |
82 | driver does not have support for XDP, or XDP_SKB is explicitly chosen | |
83 | when loading the XDP program, XDP_SKB mode is employed that uses SKBs | |
84 | together with the generic XDP support and copies out the data to user | |
85 | space. A fallback mode that works for any network device. On the other | |
86 | hand, if the driver has support for XDP, it will be used by the AF_XDP | |
87 | code to provide better performance, but there is still a copy of the | |
88 | data into user space. | |
89 | ||
90 | Concepts | |
91 | ======== | |
92 | ||
93 | In order to use an AF_XDP socket, a number of associated objects need | |
e0e4f8e9 MK |
94 | to be setup. These objects and their options are explained in the |
95 | following sections. | |
b4b8faa1 | 96 | |
e0e4f8e9 MK |
97 | For an overview on how AF_XDP works, you can also take a look at the |
98 | Linux Plumbers paper from 2018 on the subject: | |
99 | http://vger.kernel.org/lpc_net2018_talks/lpc18_paper_af_xdp_perf-v2.pdf. Do | |
100 | NOT consult the paper from 2017 on "AF_PACKET v4", the first attempt | |
101 | at AF_XDP. Nearly everything changed since then. Jonathan Corbet has | |
102 | also written an excellent article on LWN, "Accelerating networking | |
103 | with AF_XDP". It can be found at https://lwn.net/Articles/750845/. | |
b4b8faa1 MK |
104 | |
105 | UMEM | |
106 | ---- | |
107 | ||
108 | UMEM is a region of virtual contiguous memory, divided into | |
109 | equal-sized frames. An UMEM is associated to a netdev and a specific | |
bbff2f32 BT |
110 | queue id of that netdev. It is created and configured (chunk size, |
111 | headroom, start address and size) by using the XDP_UMEM_REG setsockopt | |
112 | system call. A UMEM is bound to a netdev and queue id, via the bind() | |
113 | system call. | |
b4b8faa1 MK |
114 | |
115 | An AF_XDP is socket linked to a single UMEM, but one UMEM can have | |
116 | multiple AF_XDP sockets. To share an UMEM created via one socket A, | |
117 | the next socket B can do this by setting the XDP_SHARED_UMEM flag in | |
118 | struct sockaddr_xdp member sxdp_flags, and passing the file descriptor | |
119 | of A to struct sockaddr_xdp member sxdp_shared_umem_fd. | |
120 | ||
e0e4f8e9 | 121 | The UMEM has two single-producer/single-consumer rings that are used |
b4b8faa1 MK |
122 | to transfer ownership of UMEM frames between the kernel and the |
123 | user-space application. | |
124 | ||
125 | Rings | |
126 | ----- | |
127 | ||
e0e4f8e9 | 128 | There are a four different kind of rings: FILL, COMPLETION, RX and |
b4b8faa1 MK |
129 | TX. All rings are single-producer/single-consumer, so the user-space |
130 | application need explicit synchronization of multiple | |
131 | processes/threads are reading/writing to them. | |
132 | ||
e0e4f8e9 | 133 | The UMEM uses two rings: FILL and COMPLETION. Each socket associated |
b4b8faa1 MK |
134 | with the UMEM must have an RX queue, TX queue or both. Say, that there |
135 | is a setup with four sockets (all doing TX and RX). Then there will be | |
e0e4f8e9 | 136 | one FILL ring, one COMPLETION ring, four TX rings and four RX rings. |
b4b8faa1 MK |
137 | |
138 | The rings are head(producer)/tail(consumer) based rings. A producer | |
139 | writes the data ring at the index pointed out by struct xdp_ring | |
140 | producer member, and increasing the producer index. A consumer reads | |
141 | the data ring at the index pointed out by struct xdp_ring consumer | |
142 | member, and increasing the consumer index. | |
143 | ||
144 | The rings are configured and created via the _RING setsockopt system | |
145 | calls and mmapped to user-space using the appropriate offset to mmap() | |
146 | (XDP_PGOFF_RX_RING, XDP_PGOFF_TX_RING, XDP_UMEM_PGOFF_FILL_RING and | |
147 | XDP_UMEM_PGOFF_COMPLETION_RING). | |
148 | ||
149 | The size of the rings need to be of size power of two. | |
150 | ||
151 | UMEM Fill Ring | |
152 | ~~~~~~~~~~~~~~ | |
153 | ||
e0e4f8e9 | 154 | The FILL ring is used to transfer ownership of UMEM frames from |
bbff2f32 BT |
155 | user-space to kernel-space. The UMEM addrs are passed in the ring. As |
156 | an example, if the UMEM is 64k and each chunk is 4k, then the UMEM has | |
157 | 16 chunks and can pass addrs between 0 and 64k. | |
b4b8faa1 MK |
158 | |
159 | Frames passed to the kernel are used for the ingress path (RX rings). | |
160 | ||
d57f172c KL |
161 | The user application produces UMEM addrs to this ring. Note that, if |
162 | running the application with aligned chunk mode, the kernel will mask | |
163 | the incoming addr. E.g. for a chunk size of 2k, the log2(2048) LSB of | |
164 | the addr will be masked off, meaning that 2048, 2050 and 3000 refers | |
165 | to the same chunk. If the user application is run in the unaligned | |
166 | chunks mode, then the incoming addr will be left untouched. | |
bbff2f32 | 167 | |
b4b8faa1 | 168 | |
7ccc4f18 KD |
169 | UMEM Completion Ring |
170 | ~~~~~~~~~~~~~~~~~~~~ | |
b4b8faa1 | 171 | |
e0e4f8e9 MK |
172 | The COMPLETION Ring is used transfer ownership of UMEM frames from |
173 | kernel-space to user-space. Just like the FILL ring, UMEM indices are | |
b4b8faa1 MK |
174 | used. |
175 | ||
176 | Frames passed from the kernel to user-space are frames that has been | |
177 | sent (TX ring) and can be used by user-space again. | |
178 | ||
bbff2f32 | 179 | The user application consumes UMEM addrs from this ring. |
b4b8faa1 MK |
180 | |
181 | ||
182 | RX Ring | |
183 | ~~~~~~~ | |
184 | ||
185 | The RX ring is the receiving side of a socket. Each entry in the ring | |
bbff2f32 BT |
186 | is a struct xdp_desc descriptor. The descriptor contains UMEM offset |
187 | (addr) and the length of the data (len). | |
b4b8faa1 | 188 | |
e0e4f8e9 | 189 | If no frames have been passed to kernel via the FILL ring, no |
b4b8faa1 MK |
190 | descriptors will (or can) appear on the RX ring. |
191 | ||
192 | The user application consumes struct xdp_desc descriptors from this | |
193 | ring. | |
194 | ||
195 | TX Ring | |
196 | ~~~~~~~ | |
197 | ||
198 | The TX ring is used to send frames. The struct xdp_desc descriptor is | |
199 | filled (index, length and offset) and passed into the ring. | |
200 | ||
201 | To start the transfer a sendmsg() system call is required. This might | |
202 | be relaxed in the future. | |
203 | ||
204 | The user application produces struct xdp_desc descriptors to this | |
205 | ring. | |
206 | ||
e0e4f8e9 MK |
207 | Libbpf |
208 | ====== | |
209 | ||
210 | Libbpf is a helper library for eBPF and XDP that makes using these | |
211 | technologies a lot simpler. It also contains specific helper functions | |
212 | in tools/lib/bpf/xsk.h for facilitating the use of AF_XDP. It | |
213 | contains two types of functions: those that can be used to make the | |
214 | setup of AF_XDP socket easier and ones that can be used in the data | |
215 | plane to access the rings safely and quickly. To see an example on how | |
216 | to use this API, please take a look at the sample application in | |
217 | samples/bpf/xdpsock_usr.c which uses libbpf for both setup and data | |
218 | plane operations. | |
219 | ||
220 | We recommend that you use this library unless you have become a power | |
221 | user. It will make your program a lot simpler. | |
222 | ||
b4b8faa1 | 223 | XSKMAP / BPF_MAP_TYPE_XSKMAP |
e0e4f8e9 | 224 | ============================ |
b4b8faa1 MK |
225 | |
226 | On XDP side there is a BPF map type BPF_MAP_TYPE_XSKMAP (XSKMAP) that | |
227 | is used in conjunction with bpf_redirect_map() to pass the ingress | |
228 | frame to a socket. | |
229 | ||
230 | The user application inserts the socket into the map, via the bpf() | |
231 | system call. | |
232 | ||
233 | Note that if an XDP program tries to redirect to a socket that does | |
234 | not match the queue configuration and netdev, the frame will be | |
235 | dropped. E.g. an AF_XDP socket is bound to netdev eth0 and | |
236 | queue 17. Only the XDP program executing for eth0 and queue 17 will | |
237 | successfully pass data to the socket. Please refer to the sample | |
238 | application (samples/bpf/) in for an example. | |
239 | ||
e0e4f8e9 MK |
240 | Configuration Flags and Socket Options |
241 | ====================================== | |
242 | ||
243 | These are the various configuration flags that can be used to control | |
244 | and monitor the behavior of AF_XDP sockets. | |
245 | ||
f35e0cc2 BS |
246 | XDP_COPY and XDP_ZEROCOPY bind flags |
247 | ------------------------------------ | |
e0e4f8e9 MK |
248 | |
249 | When you bind to a socket, the kernel will first try to use zero-copy | |
250 | copy. If zero-copy is not supported, it will fall back on using copy | |
251 | mode, i.e. copying all packets out to user space. But if you would | |
252 | like to force a certain mode, you can use the following flags. If you | |
253 | pass the XDP_COPY flag to the bind call, the kernel will force the | |
254 | socket into copy mode. If it cannot use copy mode, the bind call will | |
f35e0cc2 | 255 | fail with an error. Conversely, the XDP_ZEROCOPY flag will force the |
e0e4f8e9 MK |
256 | socket into zero-copy mode or fail. |
257 | ||
258 | XDP_SHARED_UMEM bind flag | |
259 | ------------------------- | |
260 | ||
acabf328 MK |
261 | This flag enables you to bind multiple sockets to the same UMEM. It |
262 | works on the same queue id, between queue ids and between | |
263 | netdevs/devices. In this mode, each socket has their own RX and TX | |
264 | rings as usual, but you are going to have one or more FILL and | |
265 | COMPLETION ring pairs. You have to create one of these pairs per | |
266 | unique netdev and queue id tuple that you bind to. | |
267 | ||
268 | Starting with the case were we would like to share a UMEM between | |
269 | sockets bound to the same netdev and queue id. The UMEM (tied to the | |
270 | fist socket created) will only have a single FILL ring and a single | |
271 | COMPLETION ring as there is only on unique netdev,queue_id tuple that | |
272 | we have bound to. To use this mode, create the first socket and bind | |
273 | it in the normal way. Create a second socket and create an RX and a TX | |
274 | ring, or at least one of them, but no FILL or COMPLETION rings as the | |
275 | ones from the first socket will be used. In the bind call, set he | |
e0e4f8e9 MK |
276 | XDP_SHARED_UMEM option and provide the initial socket's fd in the |
277 | sxdp_shared_umem_fd field. You can attach an arbitrary number of extra | |
278 | sockets this way. | |
279 | ||
280 | What socket will then a packet arrive on? This is decided by the XDP | |
281 | program. Put all the sockets in the XSK_MAP and just indicate which | |
282 | index in the array you would like to send each packet to. A simple | |
283 | round-robin example of distributing packets is shown below: | |
284 | ||
285 | .. code-block:: c | |
286 | ||
287 | #include <linux/bpf.h> | |
288 | #include "bpf_helpers.h" | |
289 | ||
290 | #define MAX_SOCKS 16 | |
291 | ||
292 | struct { | |
4b9718b5 IM |
293 | __uint(type, BPF_MAP_TYPE_XSKMAP); |
294 | __uint(max_entries, MAX_SOCKS); | |
295 | __uint(key_size, sizeof(int)); | |
296 | __uint(value_size, sizeof(int)); | |
e0e4f8e9 MK |
297 | } xsks_map SEC(".maps"); |
298 | ||
299 | static unsigned int rr; | |
300 | ||
301 | SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) | |
302 | { | |
4b9718b5 | 303 | rr = (rr + 1) & (MAX_SOCKS - 1); |
e0e4f8e9 | 304 | |
4b9718b5 | 305 | return bpf_redirect_map(&xsks_map, rr, XDP_DROP); |
e0e4f8e9 MK |
306 | } |
307 | ||
308 | Note, that since there is only a single set of FILL and COMPLETION | |
309 | rings, and they are single producer, single consumer rings, you need | |
310 | to make sure that multiple processes or threads do not use these rings | |
311 | concurrently. There are no synchronization primitives in the | |
312 | libbpf code that protects multiple users at this point in time. | |
313 | ||
57afa8b0 | 314 | Libbpf uses this mode if you create more than one socket tied to the |
acabf328 | 315 | same UMEM. However, note that you need to supply the |
57afa8b0 MK |
316 | XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD libbpf_flag with the |
317 | xsk_socket__create calls and load your own XDP program as there is no | |
318 | built in one in libbpf that will route the traffic for you. | |
319 | ||
acabf328 MK |
320 | The second case is when you share a UMEM between sockets that are |
321 | bound to different queue ids and/or netdevs. In this case you have to | |
322 | create one FILL ring and one COMPLETION ring for each unique | |
323 | netdev,queue_id pair. Let us say you want to create two sockets bound | |
324 | to two different queue ids on the same netdev. Create the first socket | |
325 | and bind it in the normal way. Create a second socket and create an RX | |
326 | and a TX ring, or at least one of them, and then one FILL and | |
327 | COMPLETION ring for this socket. Then in the bind call, set he | |
328 | XDP_SHARED_UMEM option and provide the initial socket's fd in the | |
329 | sxdp_shared_umem_fd field as you registered the UMEM on that | |
330 | socket. These two sockets will now share one and the same UMEM. | |
331 | ||
332 | There is no need to supply an XDP program like the one in the previous | |
333 | case where sockets were bound to the same queue id and | |
334 | device. Instead, use the NIC's packet steering capabilities to steer | |
335 | the packets to the right queue. In the previous example, there is only | |
336 | one queue shared among sockets, so the NIC cannot do this steering. It | |
337 | can only steer between queues. | |
338 | ||
339 | In libbpf, you need to use the xsk_socket__create_shared() API as it | |
340 | takes a reference to a FILL ring and a COMPLETION ring that will be | |
341 | created for you and bound to the shared UMEM. You can use this | |
342 | function for all the sockets you create, or you can use it for the | |
343 | second and following ones and use xsk_socket__create() for the first | |
344 | one. Both methods yield the same result. | |
345 | ||
346 | Note that a UMEM can be shared between sockets on the same queue id | |
347 | and device, as well as between queues on the same device and between | |
348 | devices at the same time. | |
349 | ||
e0e4f8e9 MK |
350 | XDP_USE_NEED_WAKEUP bind flag |
351 | ----------------------------- | |
352 | ||
353 | This option adds support for a new flag called need_wakeup that is | |
354 | present in the FILL ring and the TX ring, the rings for which user | |
355 | space is a producer. When this option is set in the bind call, the | |
356 | need_wakeup flag will be set if the kernel needs to be explicitly | |
357 | woken up by a syscall to continue processing packets. If the flag is | |
358 | zero, no syscall is needed. | |
359 | ||
360 | If the flag is set on the FILL ring, the application needs to call | |
361 | poll() to be able to continue to receive packets on the RX ring. This | |
362 | can happen, for example, when the kernel has detected that there are no | |
363 | more buffers on the FILL ring and no buffers left on the RX HW ring of | |
364 | the NIC. In this case, interrupts are turned off as the NIC cannot | |
365 | receive any packets (as there are no buffers to put them in), and the | |
366 | need_wakeup flag is set so that user space can put buffers on the | |
367 | FILL ring and then call poll() so that the kernel driver can put these | |
368 | buffers on the HW ring and start to receive packets. | |
369 | ||
370 | If the flag is set for the TX ring, it means that the application | |
371 | needs to explicitly notify the kernel to send any packets put on the | |
372 | TX ring. This can be accomplished either by a poll() call, as in the | |
373 | RX path, or by calling sendto(). | |
374 | ||
375 | An example of how to use this flag can be found in | |
376 | samples/bpf/xdpsock_user.c. An example with the use of libbpf helpers | |
377 | would look like this for the TX path: | |
378 | ||
379 | .. code-block:: c | |
380 | ||
381 | if (xsk_ring_prod__needs_wakeup(&my_tx_ring)) | |
4b9718b5 | 382 | sendto(xsk_socket__fd(xsk_handle), NULL, 0, MSG_DONTWAIT, NULL, 0); |
e0e4f8e9 MK |
383 | |
384 | I.e., only use the syscall if the flag is set. | |
385 | ||
386 | We recommend that you always enable this mode as it usually leads to | |
387 | better performance especially if you run the application and the | |
388 | driver on the same core, but also if you use different cores for the | |
389 | application and the kernel driver, as it reduces the number of | |
390 | syscalls needed for the TX path. | |
391 | ||
392 | XDP_{RX|TX|UMEM_FILL|UMEM_COMPLETION}_RING setsockopts | |
393 | ------------------------------------------------------ | |
394 | ||
395 | These setsockopts sets the number of descriptors that the RX, TX, | |
396 | FILL, and COMPLETION rings respectively should have. It is mandatory | |
397 | to set the size of at least one of the RX and TX rings. If you set | |
398 | both, you will be able to both receive and send traffic from your | |
399 | application, but if you only want to do one of them, you can save | |
400 | resources by only setting up one of them. Both the FILL ring and the | |
57afa8b0 MK |
401 | COMPLETION ring are mandatory as you need to have a UMEM tied to your |
402 | socket. But if the XDP_SHARED_UMEM flag is used, any socket after the | |
403 | first one does not have a UMEM and should in that case not have any | |
acabf328 | 404 | FILL or COMPLETION rings created as the ones from the shared UMEM will |
57afa8b0 MK |
405 | be used. Note, that the rings are single-producer single-consumer, so |
406 | do not try to access them from multiple processes at the same | |
407 | time. See the XDP_SHARED_UMEM section. | |
408 | ||
409 | In libbpf, you can create Rx-only and Tx-only sockets by supplying | |
410 | NULL to the rx and tx arguments, respectively, to the | |
411 | xsk_socket__create function. | |
412 | ||
413 | If you create a Tx-only socket, we recommend that you do not put any | |
414 | packets on the fill ring. If you do this, drivers might think you are | |
415 | going to receive something when you in fact will not, and this can | |
416 | negatively impact performance. | |
e0e4f8e9 MK |
417 | |
418 | XDP_UMEM_REG setsockopt | |
419 | ----------------------- | |
420 | ||
421 | This setsockopt registers a UMEM to a socket. This is the area that | |
a266ef69 | 422 | contain all the buffers that packet can reside in. The call takes a |
e0e4f8e9 MK |
423 | pointer to the beginning of this area and the size of it. Moreover, it |
424 | also has parameter called chunk_size that is the size that the UMEM is | |
425 | divided into. It can only be 2K or 4K at the moment. If you have an | |
426 | UMEM area that is 128K and a chunk size of 2K, this means that you | |
427 | will be able to hold a maximum of 128K / 2K = 64 packets in your UMEM | |
428 | area and that your largest packet size can be 2K. | |
429 | ||
430 | There is also an option to set the headroom of each single buffer in | |
431 | the UMEM. If you set this to N bytes, it means that the packet will | |
432 | start N bytes into the buffer leaving the first N bytes for the | |
433 | application to use. The final option is the flags field, but it will | |
434 | be dealt with in separate sections for each UMEM flag. | |
435 | ||
f7306ace IM |
436 | SO_BINDTODEVICE setsockopt |
437 | -------------------------- | |
438 | ||
439 | This is a generic SOL_SOCKET option that can be used to tie AF_XDP | |
440 | socket to a particular network interface. It is useful when a socket | |
441 | is created by a privileged process and passed to a non-privileged one. | |
442 | Once the option is set, kernel will refuse attempts to bind that socket | |
443 | to a different interface. Updating the value requires CAP_NET_RAW. | |
444 | ||
e0e4f8e9 MK |
445 | XDP_STATISTICS getsockopt |
446 | ------------------------- | |
447 | ||
448 | Gets drop statistics of a socket that can be useful for debug | |
449 | purposes. The supported statistics are shown below: | |
450 | ||
451 | .. code-block:: c | |
452 | ||
453 | struct xdp_statistics { | |
4b9718b5 IM |
454 | __u64 rx_dropped; /* Dropped for reasons other than invalid desc */ |
455 | __u64 rx_invalid_descs; /* Dropped due to invalid descriptor */ | |
456 | __u64 tx_invalid_descs; /* Dropped due to invalid descriptor */ | |
e0e4f8e9 MK |
457 | }; |
458 | ||
459 | XDP_OPTIONS getsockopt | |
460 | ---------------------- | |
461 | ||
462 | Gets options from an XDP socket. The only one supported so far is | |
463 | XDP_OPTIONS_ZEROCOPY which tells you if zero-copy is on or not. | |
464 | ||
49ca37d0 MK |
465 | Multi-Buffer Support |
466 | ==================== | |
467 | ||
468 | With multi-buffer support, programs using AF_XDP sockets can receive | |
469 | and transmit packets consisting of multiple buffers both in copy and | |
470 | zero-copy mode. For example, a packet can consist of two | |
471 | frames/buffers, one with the header and the other one with the data, | |
472 | or a 9K Ethernet jumbo frame can be constructed by chaining together | |
473 | three 4K frames. | |
474 | ||
475 | Some definitions: | |
476 | ||
477 | * A packet consists of one or more frames | |
478 | ||
479 | * A descriptor in one of the AF_XDP rings always refers to a single | |
480 | frame. In the case the packet consists of a single frame, the | |
481 | descriptor refers to the whole packet. | |
482 | ||
483 | To enable multi-buffer support for an AF_XDP socket, use the new bind | |
484 | flag XDP_USE_SG. If this is not provided, all multi-buffer packets | |
485 | will be dropped just as before. Note that the XDP program loaded also | |
486 | needs to be in multi-buffer mode. This can be accomplished by using | |
487 | "xdp.frags" as the section name of the XDP program used. | |
488 | ||
489 | To represent a packet consisting of multiple frames, a new flag called | |
490 | XDP_PKT_CONTD is introduced in the options field of the Rx and Tx | |
491 | descriptors. If it is true (1) the packet continues with the next | |
492 | descriptor and if it is false (0) it means this is the last descriptor | |
493 | of the packet. Why the reverse logic of end-of-packet (eop) flag found | |
494 | in many NICs? Just to preserve compatibility with non-multi-buffer | |
495 | applications that have this bit set to false for all packets on Rx, | |
496 | and the apps set the options field to zero for Tx, as anything else | |
497 | will be treated as an invalid descriptor. | |
498 | ||
499 | These are the semantics for producing packets onto AF_XDP Tx ring | |
500 | consisting of multiple frames: | |
501 | ||
502 | * When an invalid descriptor is found, all the other | |
503 | descriptors/frames of this packet are marked as invalid and not | |
504 | completed. The next descriptor is treated as the start of a new | |
505 | packet, even if this was not the intent (because we cannot guess | |
506 | the intent). As before, if your program is producing invalid | |
507 | descriptors you have a bug that must be fixed. | |
508 | ||
509 | * Zero length descriptors are treated as invalid descriptors. | |
510 | ||
511 | * For copy mode, the maximum supported number of frames in a packet is | |
512 | equal to CONFIG_MAX_SKB_FRAGS + 1. If it is exceeded, all | |
513 | descriptors accumulated so far are dropped and treated as | |
514 | invalid. To produce an application that will work on any system | |
515 | regardless of this config setting, limit the number of frags to 18, | |
516 | as the minimum value of the config is 17. | |
517 | ||
518 | * For zero-copy mode, the limit is up to what the NIC HW | |
519 | supports. Usually at least five on the NICs we have checked. We | |
520 | consciously chose to not enforce a rigid limit (such as | |
521 | CONFIG_MAX_SKB_FRAGS + 1) for zero-copy mode, as it would have | |
522 | resulted in copy actions under the hood to fit into what limit the | |
523 | NIC supports. Kind of defeats the purpose of zero-copy mode. How to | |
524 | probe for this limit is explained in the "probe for multi-buffer | |
525 | support" section. | |
526 | ||
527 | On the Rx path in copy-mode, the xsk core copies the XDP data into | |
528 | multiple descriptors, if needed, and sets the XDP_PKT_CONTD flag as | |
529 | detailed before. Zero-copy mode works the same, though the data is not | |
530 | copied. When the application gets a descriptor with the XDP_PKT_CONTD | |
531 | flag set to one, it means that the packet consists of multiple buffers | |
532 | and it continues with the next buffer in the following | |
533 | descriptor. When a descriptor with XDP_PKT_CONTD == 0 is received, it | |
534 | means that this is the last buffer of the packet. AF_XDP guarantees | |
535 | that only a complete packet (all frames in the packet) is sent to the | |
536 | application. If there is not enough space in the AF_XDP Rx ring, all | |
537 | frames of the packet will be dropped. | |
538 | ||
539 | If application reads a batch of descriptors, using for example the libxdp | |
540 | interfaces, it is not guaranteed that the batch will end with a full | |
541 | packet. It might end in the middle of a packet and the rest of the | |
542 | buffers of that packet will arrive at the beginning of the next batch, | |
543 | since the libxdp interface does not read the whole ring (unless you | |
544 | have an enormous batch size or a very small ring size). | |
545 | ||
546 | An example program each for Rx and Tx multi-buffer support can be found | |
547 | later in this document. | |
548 | ||
b4b8faa1 | 549 | Usage |
49ca37d0 | 550 | ----- |
b4b8faa1 | 551 | |
e0e4f8e9 | 552 | In order to use AF_XDP sockets two parts are needed. The |
b4b8faa1 MK |
553 | user-space application and the XDP program. For a complete setup and |
554 | usage example, please refer to the sample application. The user-space | |
0bed6137 EL |
555 | side is xdpsock_user.c and the XDP side is part of libbpf. |
556 | ||
e0e4f8e9 MK |
557 | The XDP code sample included in tools/lib/bpf/xsk.c is the following: |
558 | ||
559 | .. code-block:: c | |
0bed6137 EL |
560 | |
561 | SEC("xdp_sock") int xdp_sock_prog(struct xdp_md *ctx) | |
562 | { | |
563 | int index = ctx->rx_queue_index; | |
564 | ||
e0e4f8e9 | 565 | // A set entry here means that the corresponding queue_id |
0bed6137 EL |
566 | // has an active AF_XDP socket bound to it. |
567 | if (bpf_map_lookup_elem(&xsks_map, &index)) | |
568 | return bpf_redirect_map(&xsks_map, index, 0); | |
569 | ||
570 | return XDP_PASS; | |
571 | } | |
b4b8faa1 | 572 | |
e0e4f8e9 MK |
573 | A simple but not so performance ring dequeue and enqueue could look |
574 | like this: | |
575 | ||
576 | .. code-block:: c | |
b4b8faa1 | 577 | |
bbff2f32 | 578 | // struct xdp_rxtx_ring { |
4b9718b5 IM |
579 | // __u32 *producer; |
580 | // __u32 *consumer; | |
581 | // struct xdp_desc *desc; | |
bbff2f32 BT |
582 | // }; |
583 | ||
584 | // struct xdp_umem_ring { | |
4b9718b5 IM |
585 | // __u32 *producer; |
586 | // __u32 *consumer; | |
587 | // __u64 *desc; | |
bbff2f32 BT |
588 | // }; |
589 | ||
b4b8faa1 MK |
590 | // typedef struct xdp_rxtx_ring RING; |
591 | // typedef struct xdp_umem_ring RING; | |
592 | ||
593 | // typedef struct xdp_desc RING_TYPE; | |
bbff2f32 | 594 | // typedef __u64 RING_TYPE; |
b4b8faa1 MK |
595 | |
596 | int dequeue_one(RING *ring, RING_TYPE *item) | |
597 | { | |
bbff2f32 | 598 | __u32 entries = *ring->producer - *ring->consumer; |
b4b8faa1 MK |
599 | |
600 | if (entries == 0) | |
601 | return -1; | |
602 | ||
603 | // read-barrier! | |
604 | ||
bbff2f32 BT |
605 | *item = ring->desc[*ring->consumer & (RING_SIZE - 1)]; |
606 | (*ring->consumer)++; | |
b4b8faa1 MK |
607 | return 0; |
608 | } | |
609 | ||
610 | int enqueue_one(RING *ring, const RING_TYPE *item) | |
611 | { | |
bbff2f32 | 612 | u32 free_entries = RING_SIZE - (*ring->producer - *ring->consumer); |
b4b8faa1 MK |
613 | |
614 | if (free_entries == 0) | |
615 | return -1; | |
616 | ||
bbff2f32 | 617 | ring->desc[*ring->producer & (RING_SIZE - 1)] = *item; |
b4b8faa1 MK |
618 | |
619 | // write-barrier! | |
620 | ||
bbff2f32 | 621 | (*ring->producer)++; |
b4b8faa1 MK |
622 | return 0; |
623 | } | |
624 | ||
e0e4f8e9 MK |
625 | But please use the libbpf functions as they are optimized and ready to |
626 | use. Will make your life easier. | |
b4b8faa1 | 627 | |
49ca37d0 MK |
628 | Usage Multi-Buffer Rx |
629 | --------------------- | |
630 | ||
631 | Here is a simple Rx path pseudo-code example (using libxdp interfaces | |
632 | for simplicity). Error paths have been excluded to keep it short: | |
633 | ||
634 | .. code-block:: c | |
635 | ||
636 | void rx_packets(struct xsk_socket_info *xsk) | |
637 | { | |
638 | static bool new_packet = true; | |
639 | u32 idx_rx = 0, idx_fq = 0; | |
640 | static char *pkt; | |
641 | ||
642 | int rcvd = xsk_ring_cons__peek(&xsk->rx, opt_batch_size, &idx_rx); | |
643 | ||
644 | xsk_ring_prod__reserve(&xsk->umem->fq, rcvd, &idx_fq); | |
645 | ||
646 | for (int i = 0; i < rcvd; i++) { | |
647 | struct xdp_desc *desc = xsk_ring_cons__rx_desc(&xsk->rx, idx_rx++); | |
648 | char *frag = xsk_umem__get_data(xsk->umem->buffer, desc->addr); | |
649 | bool eop = !(desc->options & XDP_PKT_CONTD); | |
650 | ||
651 | if (new_packet) | |
652 | pkt = frag; | |
653 | else | |
654 | add_frag_to_pkt(pkt, frag); | |
655 | ||
656 | if (eop) | |
657 | process_pkt(pkt); | |
658 | ||
659 | new_packet = eop; | |
660 | ||
661 | *xsk_ring_prod__fill_addr(&xsk->umem->fq, idx_fq++) = desc->addr; | |
662 | } | |
663 | ||
664 | xsk_ring_prod__submit(&xsk->umem->fq, rcvd); | |
665 | xsk_ring_cons__release(&xsk->rx, rcvd); | |
666 | } | |
667 | ||
668 | Usage Multi-Buffer Tx | |
669 | --------------------- | |
670 | ||
671 | Here is an example Tx path pseudo-code (using libxdp interfaces for | |
672 | simplicity) ignoring that the umem is finite in size, and that we | |
673 | eventually will run out of packets to send. Also assumes pkts.addr | |
674 | points to a valid location in the umem. | |
675 | ||
676 | .. code-block:: c | |
677 | ||
678 | void tx_packets(struct xsk_socket_info *xsk, struct pkt *pkts, | |
679 | int batch_size) | |
680 | { | |
681 | u32 idx, i, pkt_nb = 0; | |
682 | ||
683 | xsk_ring_prod__reserve(&xsk->tx, batch_size, &idx); | |
684 | ||
685 | for (i = 0; i < batch_size;) { | |
686 | u64 addr = pkts[pkt_nb].addr; | |
687 | u32 len = pkts[pkt_nb].size; | |
688 | ||
689 | do { | |
690 | struct xdp_desc *tx_desc; | |
691 | ||
692 | tx_desc = xsk_ring_prod__tx_desc(&xsk->tx, idx + i++); | |
693 | tx_desc->addr = addr; | |
694 | ||
695 | if (len > xsk_frame_size) { | |
696 | tx_desc->len = xsk_frame_size; | |
697 | tx_desc->options = XDP_PKT_CONTD; | |
698 | } else { | |
699 | tx_desc->len = len; | |
700 | tx_desc->options = 0; | |
701 | pkt_nb++; | |
702 | } | |
703 | len -= tx_desc->len; | |
704 | addr += xsk_frame_size; | |
705 | ||
706 | if (i == batch_size) { | |
707 | /* Remember len, addr, pkt_nb for next iteration. | |
708 | * Skipped for simplicity. | |
709 | */ | |
710 | break; | |
711 | } | |
712 | } while (len); | |
713 | } | |
714 | ||
715 | xsk_ring_prod__submit(&xsk->tx, i); | |
716 | } | |
717 | ||
718 | Probing for Multi-Buffer Support | |
719 | -------------------------------- | |
720 | ||
721 | To discover if a driver supports multi-buffer AF_XDP in SKB or DRV | |
722 | mode, use the XDP_FEATURES feature of netlink in linux/netdev.h to | |
723 | query for NETDEV_XDP_ACT_RX_SG support. This is the same flag as for | |
724 | querying for XDP multi-buffer support. If XDP supports multi-buffer in | |
725 | a driver, then AF_XDP will also support that in SKB and DRV mode. | |
726 | ||
727 | To discover if a driver supports multi-buffer AF_XDP in zero-copy | |
728 | mode, use XDP_FEATURES and first check the NETDEV_XDP_ACT_XSK_ZEROCOPY | |
729 | flag. If it is set, it means that at least zero-copy is supported and | |
730 | you should go and check the netlink attribute | |
731 | NETDEV_A_DEV_XDP_ZC_MAX_SEGS in linux/netdev.h. An unsigned integer | |
732 | value will be returned stating the max number of frags that are | |
733 | supported by this device in zero-copy mode. These are the possible | |
734 | return values: | |
735 | ||
736 | 1: Multi-buffer for zero-copy is not supported by this device, as max | |
737 | one fragment supported means that multi-buffer is not possible. | |
738 | ||
739 | >=2: Multi-buffer is supported in zero-copy mode for this device. The | |
740 | returned number signifies the max number of frags supported. | |
741 | ||
742 | For an example on how these are used through libbpf, please take a | |
743 | look at tools/testing/selftests/bpf/xskxceiver.c. | |
744 | ||
745 | Multi-Buffer Support for Zero-Copy Drivers | |
746 | ------------------------------------------ | |
747 | ||
748 | Zero-copy drivers usually use the batched APIs for Rx and Tx | |
749 | processing. Note that the Tx batch API guarantees that it will provide | |
750 | a batch of Tx descriptors that ends with full packet at the end. This | |
751 | to facilitate extending a zero-copy driver with multi-buffer support. | |
752 | ||
b4b8faa1 MK |
753 | Sample application |
754 | ================== | |
755 | ||
756 | There is a xdpsock benchmarking/test application included that | |
e0e4f8e9 MK |
757 | demonstrates how to use AF_XDP sockets with private UMEMs. Say that |
758 | you would like your UDP traffic from port 4242 to end up in queue 16, | |
759 | that we will enable AF_XDP on. Here, we use ethtool for this:: | |
b4b8faa1 MK |
760 | |
761 | ethtool -N p3p2 rx-flow-hash udp4 fn | |
762 | ethtool -N p3p2 flow-type udp4 src-port 4242 dst-port 4242 \ | |
763 | action 16 | |
764 | ||
765 | Running the rxdrop benchmark in XDP_DRV mode can then be done | |
766 | using:: | |
767 | ||
768 | samples/bpf/xdpsock -i p3p2 -q 16 -r -N | |
769 | ||
770 | For XDP_SKB mode, use the switch "-S" instead of "-N" and all options | |
771 | can be displayed with "-h", as usual. | |
772 | ||
e0e4f8e9 MK |
773 | This sample application uses libbpf to make the setup and usage of |
774 | AF_XDP simpler. If you want to know how the raw uapi of AF_XDP is | |
775 | really used to make something more advanced, take a look at the libbpf | |
776 | code in tools/lib/bpf/xsk.[ch]. | |
777 | ||
0f4a9b7d MK |
778 | FAQ |
779 | ======= | |
780 | ||
781 | Q: I am not seeing any traffic on the socket. What am I doing wrong? | |
782 | ||
783 | A: When a netdev of a physical NIC is initialized, Linux usually | |
e0e4f8e9 | 784 | allocates one RX and TX queue pair per core. So on a 8 core system, |
0f4a9b7d MK |
785 | queue ids 0 to 7 will be allocated, one per core. In the AF_XDP |
786 | bind call or the xsk_socket__create libbpf function call, you | |
787 | specify a specific queue id to bind to and it is only the traffic | |
788 | towards that queue you are going to get on you socket. So in the | |
789 | example above, if you bind to queue 0, you are NOT going to get any | |
790 | traffic that is distributed to queues 1 through 7. If you are | |
791 | lucky, you will see the traffic, but usually it will end up on one | |
792 | of the queues you have not bound to. | |
793 | ||
794 | There are a number of ways to solve the problem of getting the | |
795 | traffic you want to the queue id you bound to. If you want to see | |
796 | all the traffic, you can force the netdev to only have 1 queue, queue | |
797 | id 0, and then bind to queue 0. You can use ethtool to do this:: | |
798 | ||
221fb726 | 799 | sudo ethtool -L <interface> combined 1 |
0f4a9b7d MK |
800 | |
801 | If you want to only see part of the traffic, you can program the | |
802 | NIC through ethtool to filter out your traffic to a single queue id | |
803 | that you can bind your XDP socket to. Here is one example in which | |
804 | UDP traffic to and from port 4242 are sent to queue 2:: | |
805 | ||
221fb726 RD |
806 | sudo ethtool -N <interface> rx-flow-hash udp4 fn |
807 | sudo ethtool -N <interface> flow-type udp4 src-port 4242 dst-port \ | |
808 | 4242 action 2 | |
0f4a9b7d | 809 | |
e0e4f8e9 | 810 | A number of other ways are possible all up to the capabilities of |
0f4a9b7d MK |
811 | the NIC you have. |
812 | ||
a266ef69 | 813 | Q: Can I use the XSKMAP to implement a switch between different umems |
e0e4f8e9 MK |
814 | in copy mode? |
815 | ||
816 | A: The short answer is no, that is not supported at the moment. The | |
817 | XSKMAP can only be used to switch traffic coming in on queue id X | |
818 | to sockets bound to the same queue id X. The XSKMAP can contain | |
819 | sockets bound to different queue ids, for example X and Y, but only | |
820 | traffic goming in from queue id Y can be directed to sockets bound | |
821 | to the same queue id Y. In zero-copy mode, you should use the | |
822 | switch, or other distribution mechanism, in your NIC to direct | |
823 | traffic to the correct queue id and socket. | |
824 | ||
acabf328 MK |
825 | Q: My packets are sometimes corrupted. What is wrong? |
826 | ||
827 | A: Care has to be taken not to feed the same buffer in the UMEM into | |
828 | more than one ring at the same time. If you for example feed the | |
829 | same buffer into the FILL ring and the TX ring at the same time, the | |
830 | NIC might receive data into the buffer at the same time it is | |
831 | sending it. This will cause some packets to become corrupted. Same | |
832 | thing goes for feeding the same buffer into the FILL rings | |
833 | belonging to different queue ids or netdevs bound with the | |
834 | XDP_SHARED_UMEM flag. | |
835 | ||
b4b8faa1 MK |
836 | Credits |
837 | ======= | |
838 | ||
839 | - Björn Töpel (AF_XDP core) | |
840 | - Magnus Karlsson (AF_XDP core) | |
841 | - Alexander Duyck | |
842 | - Alexei Starovoitov | |
843 | - Daniel Borkmann | |
844 | - Jesper Dangaard Brouer | |
845 | - John Fastabend | |
846 | - Jonathan Corbet (LWN coverage) | |
847 | - Michael S. Tsirkin | |
848 | - Qi Z Zhang | |
849 | - Willem de Bruijn |