1 * Texas Instruments CPSW ethernet driver
3 Multiqueue & CBS & MQPRIO
4 =====================================================================
5 =====================================================================
7 The cpsw has 3 CBS shapers for each external ports. This document
8 describes MQPRIO and CBS Qdisc offload configuration for cpsw driver
9 based on examples. It potentially can be used in audio video bridging
10 (AVB) and time sensitive networking (TSN).
12 The following examples were tested on AM572x EVM and BBB boards.
17 Under consideration two examples with AM572x EVM running cpsw driver
20 Several prerequisites:
21 - TX queues must be rated starting from txq0 that has highest priority
22 - Traffic classes are used starting from 0, that has highest priority
23 - CBS shapers should be used with rated queues
24 - The bandwidth for CBS shapers has to be set a little bit more then
25 potential incoming rate, thus, rate of all incoming tx queues has
27 - Real rates can differ, due to discreetness
28 - Map skb-priority to txq is not enough, also skb-priority to l2 prio
29 map has to be created with ip or vconfig tool
30 - Any l2/socket prio (0 - 7) for classes can be used, but for
31 simplicity default values are used: 3 and 2
32 - only 2 classes tested: A and B, but checked and can work with more,
33 maximum allowed 4, but only for 3 rate can be set.
35 Test setup for examples
36 =======================
37 +-------------------------------+
40 |E | MAC 18:03:73:66:87:42 |
41 +-----------------------------+ +--|t | |
42 | | 1 | E | | |h |./tsn_listener -d \ |
43 | Target board: | 0 | t |--+ |0 | 18:03:73:66:87:42 -i eth0 \|
44 | AM572x EVM | 0 | h | | | -s 1500 |
46 | Only 2 classes: |Mb +---| +-------------------------------+
47 | class A, class B | |
48 | | +---| +-------------------------------+
50 | | 0 | t | | | Workstation1 |
51 | | 0 | h |--+ |E | MAC 20:cf:30:85:7d:fd |
53 +-----------------------------+ |h |./tsn_listener -d \ |
54 |0 | 20:cf:30:85:7d:fd -i eth0 \|
57 +-------------------------------+
59 *********************************************************************
60 *********************************************************************
61 *********************************************************************
62 Example 1: One port tx AVB configuration scheme for target board
63 ----------------------------------------------------------------------
64 (prints and scheme for AM572x evm, applicable for single port boards)
72 +------------------------------------------------------------------+ u
73 | +---------------+ +---------------+ +------+ +------+ | s
75 | | App 1 | | App 2 | | Apps | | Apps | | r
76 | | Class A | | Class B | | Rest | | Rest | |
77 | | Eth0 | | Eth0 | | Eth0 | | Eth1 | | s
78 | | VLAN100 | | VLAN100 | | | | | | | | p
79 | | 40 Mb/s | | 20 Mb/s | | | | | | | | a
80 | | SO_PRIORITY=3 | | SO_PRIORITY=2 | | | | | | | | c
81 | | | | | | | | | | | | | | e
82 | +---|-----------+ +---|-----------+ +---|--+ +---|--+ |
83 +-----|------------------|------------------|--------|-------------+
84 +-+ +------------+ | |
85 | | +-----------------+ +--+
87 +---|-------|-------------|-----------------------|----------------+
88 | +----+ +----+ +----+ +----+ +----+ |
89 | | p3 | | p2 | | p1 | | p0 | | p0 | | k
90 | \ / \ / \ / \ / \ / | e
91 | \ / \ / \ / \ / \ / | r
96 | +----+ +----+ +----+ +----+ | s
97 | |tc0 | |tc1 | |tc2 | |tc0 | | p
105 | +----+ +----+ +----+ +----+ +----+ |
106 | |txq0| |txq1| |txq2| |txq3| |txq4| |
107 | \ / \ / \ / \ / \ / |
108 | \ / \ / \ / \ / \ / |
110 | +-|------|------|------|--+ +--|--------------+ |
111 | | | | | | | Eth0.100 | | Eth1 | |
112 +---|------|------|------|------------------------|----------------+
115 3 2 0-1, 4-7 <- L2 priority |
118 +---|------|------|------|------------------------|----------------+
119 | | | | | |----------+ |
120 | +----+ +----+ +----+ +----+ +----+ |
121 | |dma7| |dma6| |dma5| |dma4| |dma3| |
122 | \ / \ / \ / \ / \ / | c
123 | \S / \S / \ / \ / \ / | p
128 | +----+ +----+ +----+p p+----+ | r
129 | | | | | | |o o| | | i
130 | | f3 | | f2 | | f0 |r r| f0 | | v
131 | |tc0 | |tc1 | |tc2 |t t|tc0 | | e
132 | \CBS / \CBS / \CBS /1 2\CBS / | r
133 | \S / \S / \ / \ / |
135 +------------------------------------------------------------------+
136 ========================================Eth==========================>
139 // Add 4 tx queues, for interface Eth0, and 1 tx queue for Eth1
140 $ ethtool -L eth0 rx 1 tx 5
141 rx unmodified, ignoring
144 // Check if num of queues is set correctly:
146 Channel parameters for eth0:
152 Current hardware settings:
159 // TX queues must be rated starting from 0, so set bws for tx0 and tx1
160 // Set rates 40 and 20 Mb/s appropriately.
161 // Pay attention, real speed can differ a bit due to discreetness.
162 // Leave last 2 tx queues not rated.
163 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
164 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
167 // Check maximum rate of tx (cpdma) queues:
168 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
176 // Map skb->priority to traffic class:
177 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
178 // Map traffic class to transmit queue:
179 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq2, txq3)
180 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
181 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@2 hw 1
184 // As two interface sharing same set of tx queues, assign all traffic
185 // coming to interface Eth1 to separate queue in order to not mix it
186 // with traffic from interface Eth0, so use separate txq to send
187 // packets to Eth1, so all prio -> tc0 and tc0 -> txq4
188 // Here hw 0, so here still default configuration for eth1 in hw
189 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 1 \
190 map 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 queues 1@4 hw 0
193 // Check classes settings
194 $ tc -g class show dev eth0
195 +---(100:ffe2) mqprio
199 +---(100:ffe1) mqprio
202 +---(100:ffe0) mqprio
205 $ tc -g class show dev eth1
206 +---(100:ffe0) mqprio
210 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc
211 // Set it +1 Mb for reserve (important!)
212 // here only idle slope is important, others arg are ignored
213 // Pay attention, real speed can differ a bit due to discreetness
214 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1438 \
215 hicredit 62 sendslope -959000 idleslope 41000 offload 1
216 net eth0: set FIFO3 bw = 50
219 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc:
220 // Set it +1 Mb for reserve (important!)
221 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1468 \
222 hicredit 65 sendslope -979000 idleslope 21000 offload 1
223 net eth0: set FIFO2 bw = 30
226 // Create vlan 100 to map sk->priority to vlan qos
227 $ ip link add link eth0 name eth0.100 type vlan id 100
228 8021q: 802.1Q VLAN Support v1.8
229 8021q: adding VLAN 0 to HW filter on device eth0
230 8021q: adding VLAN 0 to HW filter on device eth1
231 net eth0: Adding vlanid 100 to vlan filter
234 // Map skb->priority to L2 prio, 1 to 1
235 $ ip link set eth0.100 type vlan \
236 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
239 // Check egress map for vlan 100
240 $ cat /proc/net/vlan/eth0.100
242 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
243 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
246 // Run your appropriate tools with socket option "SO_PRIORITY"
247 // to 3 for class A and/or to 2 for class B
248 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
249 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
250 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
253 // run your listener on workstation (should be in same vlan)
254 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
255 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
256 Receiving data rate: 39012 kbps
257 Receiving data rate: 39012 kbps
258 Receiving data rate: 39012 kbps
259 Receiving data rate: 39012 kbps
260 Receiving data rate: 39012 kbps
261 Receiving data rate: 39012 kbps
262 Receiving data rate: 39012 kbps
263 Receiving data rate: 39012 kbps
264 Receiving data rate: 39012 kbps
265 Receiving data rate: 39012 kbps
266 Receiving data rate: 39012 kbps
267 Receiving data rate: 39012 kbps
268 Receiving data rate: 39000 kbps
271 // Restore default configuration if needed
272 $ ip link del eth0.100
273 $ tc qdisc del dev eth1 root
274 $ tc qdisc del dev eth0 root
275 net eth0: Prev FIFO2 is shaped
276 net eth0: set FIFO3 bw = 0
277 net eth0: set FIFO2 bw = 0
278 $ ethtool -L eth0 rx 1 tx 1
280 *********************************************************************
281 *********************************************************************
282 *********************************************************************
283 Example 2: Two port tx AVB configuration scheme for target board
284 ----------------------------------------------------------------------
285 (prints and scheme for AM572x evm, for dual emac boards only)
287 +------------------------------------------------------------------+ u
288 | +----------+ +----------+ +------+ +----------+ +----------+ | s
289 | | | | | | | | | | | | e
290 | | App 1 | | App 2 | | Apps | | App 3 | | App 4 | | r
291 | | Class A | | Class B | | Rest | | Class B | | Class A | |
292 | | Eth0 | | Eth0 | | | | | Eth1 | | Eth1 | | s
293 | | VLAN100 | | VLAN100 | | | | | VLAN100 | | VLAN100 | | p
294 | | 40 Mb/s | | 20 Mb/s | | | | | 10 Mb/s | | 30 Mb/s | | a
295 | | SO_PRI=3 | | SO_PRI=2 | | | | | SO_PRI=3 | | SO_PRI=2 | | c
296 | | | | | | | | | | | | | | | | | e
297 | +---|------+ +---|------+ +---|--+ +---|------+ +---|------+ |
298 +-----|-------------|-------------|---------|-------------|--------+
299 +-+ +-------+ | +----------+ +----+
300 | | +-------+------+ | |
302 +---|-------|-------------|--------------|-------------|-------|---+
303 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
304 | | p3 | | p2 | | p1 | | p0 | | p0 | | p1 | | p2 | | p3 | | k
305 | \ / \ / \ / \ / \ / \ / \ / \ / | e
306 | \ / \ / \ / \ / \ / \ / \ / \ / | r
307 | \/ \/ \/ \/ \/ \/ \/ \/ | n
309 | | | +----+ +----+ | | | l
311 | +----+ +----+ +----+ +----+ +----+ +----+ | s
312 | |tc0 | |tc1 | |tc2 | |tc2 | |tc1 | |tc0 | | p
313 | \ / \ / \ / \ / \ / \ / | a
314 | \ / \ / \ / \ / \ / \ / | c
315 | \/ \/ \/ \/ \/ \/ | e
316 | | | +-----+ +-----+ | | |
319 | | | | | E E | | | | |
320 | +----+ +----+ +----+ +----+ t t +----+ +----+ +----+ +----+ |
321 | |txq0| |txq1| |txq4| |txq5| h h |txq6| |txq7| |txq3| |txq2| |
322 | \ / \ / \ / \ / 0 1 \ / \ / \ / \ / |
323 | \ / \ / \ / \ / . . \ / \ / \ / \ / |
324 | \/ \/ \/ \/ 1 1 \/ \/ \/ \/ |
325 | +-|------|------|------|--+ 0 0 +-|------|------|------|--+ |
326 | | | | | | | 0 0 | | | | | | |
327 +---|------|------|------|---------------|------|------|------|----+
330 3 2 0-1, 4-7 <-L2 pri-> 0-1, 4-7 2 3
333 +---|------|------|------|---------------|------|------|------|----+
335 | +----+ +----+ +----+ +----+ +----+ +----+ +----+ +----+ |
336 | |dma7| |dma6| |dma3| |dma2| |dma1| |dma0| |dma4| |dma5| |
337 | \ / \ / \ / \ / \ / \ / \ / \ / | c
338 | \S / \S / \ / \ / \ / \ / \S / \S / | p
339 | \/ \/ \/ \/ \/ \/ \/ \/ | s
340 | | | | +----- | | | | | w
341 | | | | | +----+ | | | |
342 | | | | | | | | | | d
343 | +----+ +----+ +----+p p+----+ +----+ +----+ | r
344 | | | | | | |o o| | | | | | | i
345 | | f3 | | f2 | | f0 |r CPSW r| f3 | | f2 | | f0 | | v
346 | |tc0 | |tc1 | |tc2 |t t|tc0 | |tc1 | |tc2 | | e
347 | \CBS / \CBS / \CBS /1 2\CBS / \CBS / \CBS / | r
348 | \S / \S / \ / \S / \S / \ / |
349 | \/ \/ \/ \/ \/ \/ |
350 +------------------------------------------------------------------+
351 ========================================Eth==========================>
354 // Add 8 tx queues, for interface Eth0, but they are common, so are accessed
355 // by two interfaces Eth0 and Eth1.
356 $ ethtool -L eth1 rx 1 tx 8
357 rx unmodified, ignoring
360 // Check if num of queues is set correctly:
362 Channel parameters for eth0:
368 Current hardware settings:
375 // TX queues must be rated starting from 0, so set bws for tx0 and tx1 for Eth0
376 // and for tx2 and tx3 for Eth1. That is, rates 40 and 20 Mb/s appropriately
377 // for Eth0 and 30 and 10 Mb/s for Eth1.
378 // Real speed can differ a bit due to discreetness
379 // Leave last 4 tx queues as not rated
380 $ echo 40 > /sys/class/net/eth0/queues/tx-0/tx_maxrate
381 $ echo 20 > /sys/class/net/eth0/queues/tx-1/tx_maxrate
382 $ echo 30 > /sys/class/net/eth1/queues/tx-2/tx_maxrate
383 $ echo 10 > /sys/class/net/eth1/queues/tx-3/tx_maxrate
386 // Check maximum rate of tx (cpdma) queues:
387 $ cat /sys/class/net/eth0/queues/tx-*/tx_maxrate
398 // Map skb->priority to traffic class for Eth0:
399 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
400 // Map traffic class to transmit queue:
401 // tc0 -> txq0, tc1 -> txq1, tc2 -> (txq4, txq5)
402 $ tc qdisc replace dev eth0 handle 100: parent root mqprio num_tc 3 \
403 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@0 1@1 2@4 hw 1
406 // Check classes settings
407 $ tc -g class show dev eth0
408 +---(100:ffe2) mqprio
412 +---(100:ffe1) mqprio
415 +---(100:ffe0) mqprio
419 // Set rate for class A - 41 Mbit (tc0, txq0) using CBS Qdisc for Eth0
420 // here only idle slope is important, others ignored
421 // Real speed can differ a bit due to discreetness
422 $ tc qdisc add dev eth0 parent 100:1 cbs locredit -1470 \
423 hicredit 62 sendslope -959000 idleslope 41000 offload 1
424 net eth0: set FIFO3 bw = 50
427 // Set rate for class B - 21 Mbit (tc1, txq1) using CBS Qdisc for Eth0
428 $ tc qdisc add dev eth0 parent 100:2 cbs locredit -1470 \
429 hicredit 65 sendslope -979000 idleslope 21000 offload 1
430 net eth0: set FIFO2 bw = 30
433 // Create vlan 100 to map sk->priority to vlan qos for Eth0
434 $ ip link add link eth0 name eth0.100 type vlan id 100
435 net eth0: Adding vlanid 100 to vlan filter
438 // Map skb->priority to L2 prio for Eth0.100, one to one
439 $ ip link set eth0.100 type vlan \
440 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
443 // Check egress map for vlan 100
444 $ cat /proc/net/vlan/eth0.100
446 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
447 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
450 // Map skb->priority to traffic class for Eth1:
451 // 3pri -> tc0, 2pri -> tc1, (0,1,4-7)pri -> tc2
452 // Map traffic class to transmit queue:
453 // tc0 -> txq2, tc1 -> txq3, tc2 -> (txq6, txq7)
454 $ tc qdisc replace dev eth1 handle 100: parent root mqprio num_tc 3 \
455 map 2 2 1 0 2 2 2 2 2 2 2 2 2 2 2 2 queues 1@2 1@3 2@6 hw 1
458 // Check classes settings
459 $ tc -g class show dev eth1
460 +---(100:ffe2) mqprio
464 +---(100:ffe1) mqprio
467 +---(100:ffe0) mqprio
471 // Set rate for class A - 31 Mbit (tc0, txq2) using CBS Qdisc for Eth1
472 // here only idle slope is important, others ignored, but calculated
473 // for interface speed - 100Mb for eth1 port.
474 // Set it +1 Mb for reserve (important!)
475 $ tc qdisc add dev eth1 parent 100:3 cbs locredit -1035 \
476 hicredit 465 sendslope -69000 idleslope 31000 offload 1
477 net eth1: set FIFO3 bw = 31
480 // Set rate for class B - 11 Mbit (tc1, txq3) using CBS Qdisc for Eth1
481 // Set it +1 Mb for reserve (important!)
482 $ tc qdisc add dev eth1 parent 100:4 cbs locredit -1335 \
483 hicredit 405 sendslope -89000 idleslope 11000 offload 1
484 net eth1: set FIFO2 bw = 11
487 // Create vlan 100 to map sk->priority to vlan qos for Eth1
488 $ ip link add link eth1 name eth1.100 type vlan id 100
489 net eth1: Adding vlanid 100 to vlan filter
492 // Map skb->priority to L2 prio for Eth1.100, one to one
493 $ ip link set eth1.100 type vlan \
494 egress 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
497 // Check egress map for vlan 100
498 $ cat /proc/net/vlan/eth1.100
500 INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
501 EGRESS priority mappings: 0:0 1:1 2:2 3:3 4:4 5:5 6:6 7:7
504 // Run appropriate tools with socket option "SO_PRIORITY" to 3
505 // for class A and to 2 for class B. For both interfaces
506 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p2 -s 1500&
507 ./tsn_talker -d 18:03:73:66:87:42 -i eth0.100 -p3 -s 1500&
508 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p2 -s 1500&
509 ./tsn_talker -d 20:cf:30:85:7d:fd -i eth1.100 -p3 -s 1500&
512 // run your listener on workstation (should be in same vlan)
513 // (I took at https://www.spinics.net/lists/netdev/msg460869.html)
514 ./tsn_listener -d 18:03:73:66:87:42 -i enp5s0 -s 1500
515 Receiving data rate: 39012 kbps
516 Receiving data rate: 39012 kbps
517 Receiving data rate: 39012 kbps
518 Receiving data rate: 39012 kbps
519 Receiving data rate: 39012 kbps
520 Receiving data rate: 39012 kbps
521 Receiving data rate: 39012 kbps
522 Receiving data rate: 39012 kbps
523 Receiving data rate: 39012 kbps
524 Receiving data rate: 39012 kbps
525 Receiving data rate: 39012 kbps
526 Receiving data rate: 39012 kbps
527 Receiving data rate: 39000 kbps
530 // Restore default configuration if needed
531 $ ip link del eth1.100
532 $ ip link del eth0.100
533 $ tc qdisc del dev eth1 root
534 net eth1: Prev FIFO2 is shaped
535 net eth1: set FIFO3 bw = 0
536 net eth1: set FIFO2 bw = 0
537 $ tc qdisc del dev eth0 root
538 net eth0: Prev FIFO2 is shaped
539 net eth0: set FIFO3 bw = 0
540 net eth0: set FIFO2 bw = 0
541 $ ethtool -L eth0 rx 1 tx 1