1 .. SPDX-License-Identifier: GPL-2.0
3 =======================================
4 DSA switch configuration from userspace
5 =======================================
7 The DSA switch configuration is not integrated into the main userspace
8 network configuration suites by now and has to be performed manually.
10 .. _dsa-config-showcases:
12 Configuration showcases
13 -----------------------
15 To configure a DSA switch a couple of commands need to be executed. In this
16 documentation some common configuration scenarios are handled as showcases:
19 Every switch port acts as a different configurable Ethernet port
22 Every switch port is part of one configurable Ethernet bridge
25 Every switch port except one upstream port is part of a configurable
27 The upstream port acts as different configurable Ethernet port.
29 All configurations are performed with tools from iproute2, which is available
30 at https://www.kernel.org/pub/linux/utils/net/iproute2/
32 Through DSA every port of a switch is handled like a normal linux Ethernet
33 interface. The CPU port is the switch port connected to an Ethernet MAC chip.
34 The corresponding linux Ethernet interface is called the master interface.
35 All other corresponding linux interfaces are called slave interfaces.
37 The slave interfaces depend on the master interface being up in order for them
38 to send or receive traffic. Prior to kernel v5.12, the state of the master
39 interface had to be managed explicitly by the user. Starting with kernel v5.12,
40 the behavior is as follows:
42 - when a DSA slave interface is brought up, the master interface is
43 automatically brought up.
44 - when the master interface is brought down, all DSA slave interfaces are
45 automatically brought down.
47 In this documentation the following Ethernet interfaces are used:
53 another master interface
59 another slave interface
62 a third slave interface
65 A slave interface dedicated for upstream traffic
67 Further Ethernet interfaces can be configured similar.
68 The configured IPs and networks are:
71 * lan1: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
72 * lan2: 192.0.2.5/30 (192.0.2.4 - 192.0.2.7)
73 * lan3: 192.0.2.9/30 (192.0.2.8 - 192.0.2.11)
76 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
79 * br0: 192.0.2.129/25 (192.0.2.128 - 192.0.2.255)
80 * wan: 192.0.2.1/30 (192.0.2.0 - 192.0.2.3)
82 .. _dsa-tagged-configuration:
84 Configuration with tagging support
85 ----------------------------------
87 The tagging based configuration is desired and supported by the majority of
88 DSA switches. These switches are capable to tag incoming and outgoing traffic
89 without using a VLAN based configuration.
94 # configure each interface
95 ip addr add 192.0.2.1/30 dev lan1
96 ip addr add 192.0.2.5/30 dev lan2
97 ip addr add 192.0.2.9/30 dev lan3
99 # For kernels earlier than v5.12, the master interface needs to be
100 # brought up manually before the slave ports.
103 # bring up the slave interfaces
111 # For kernels earlier than v5.12, the master interface needs to be
112 # brought up manually before the slave ports.
115 # bring up the slave interfaces
121 ip link add name br0 type bridge
123 # add ports to bridge
124 ip link set dev lan1 master br0
125 ip link set dev lan2 master br0
126 ip link set dev lan3 master br0
128 # configure the bridge
129 ip addr add 192.0.2.129/25 dev br0
131 # bring up the bridge
132 ip link set dev br0 up
137 # For kernels earlier than v5.12, the master interface needs to be
138 # brought up manually before the slave ports.
141 # bring up the slave interfaces
146 # configure the upstream port
147 ip addr add 192.0.2.1/30 dev wan
150 ip link add name br0 type bridge
152 # add ports to bridge
153 ip link set dev lan1 master br0
154 ip link set dev lan2 master br0
156 # configure the bridge
157 ip addr add 192.0.2.129/25 dev br0
159 # bring up the bridge
160 ip link set dev br0 up
162 .. _dsa-vlan-configuration:
164 Configuration without tagging support
165 -------------------------------------
167 A minority of switches are not capable to use a taging protocol
168 (DSA_TAG_PROTO_NONE). These switches can be configured by a VLAN based
172 The configuration can only be set up via VLAN tagging and bridge setup.
176 # tag traffic on CPU port
177 ip link add link eth0 name eth0.1 type vlan id 1
178 ip link add link eth0 name eth0.2 type vlan id 2
179 ip link add link eth0 name eth0.3 type vlan id 3
181 # For kernels earlier than v5.12, the master interface needs to be
182 # brought up manually before the slave ports.
184 ip link set eth0.1 up
185 ip link set eth0.2 up
186 ip link set eth0.3 up
188 # bring up the slave interfaces
194 ip link add name br0 type bridge
196 # activate VLAN filtering
197 ip link set dev br0 type bridge vlan_filtering 1
199 # add ports to bridges
200 ip link set dev lan1 master br0
201 ip link set dev lan2 master br0
202 ip link set dev lan3 master br0
204 # tag traffic on ports
205 bridge vlan add dev lan1 vid 1 pvid untagged
206 bridge vlan add dev lan2 vid 2 pvid untagged
207 bridge vlan add dev lan3 vid 3 pvid untagged
209 # configure the VLANs
210 ip addr add 192.0.2.1/30 dev eth0.1
211 ip addr add 192.0.2.5/30 dev eth0.2
212 ip addr add 192.0.2.9/30 dev eth0.3
214 # bring up the bridge devices
221 # tag traffic on CPU port
222 ip link add link eth0 name eth0.1 type vlan id 1
224 # For kernels earlier than v5.12, the master interface needs to be
225 # brought up manually before the slave ports.
227 ip link set eth0.1 up
229 # bring up the slave interfaces
235 ip link add name br0 type bridge
237 # activate VLAN filtering
238 ip link set dev br0 type bridge vlan_filtering 1
240 # add ports to bridge
241 ip link set dev lan1 master br0
242 ip link set dev lan2 master br0
243 ip link set dev lan3 master br0
244 ip link set eth0.1 master br0
246 # tag traffic on ports
247 bridge vlan add dev lan1 vid 1 pvid untagged
248 bridge vlan add dev lan2 vid 1 pvid untagged
249 bridge vlan add dev lan3 vid 1 pvid untagged
251 # configure the bridge
252 ip addr add 192.0.2.129/25 dev br0
254 # bring up the bridge
255 ip link set dev br0 up
260 # tag traffic on CPU port
261 ip link add link eth0 name eth0.1 type vlan id 1
262 ip link add link eth0 name eth0.2 type vlan id 2
264 # For kernels earlier than v5.12, the master interface needs to be
265 # brought up manually before the slave ports.
267 ip link set eth0.1 up
268 ip link set eth0.2 up
270 # bring up the slave interfaces
276 ip link add name br0 type bridge
278 # activate VLAN filtering
279 ip link set dev br0 type bridge vlan_filtering 1
281 # add ports to bridges
282 ip link set dev wan master br0
283 ip link set eth0.1 master br0
284 ip link set dev lan1 master br0
285 ip link set dev lan2 master br0
287 # tag traffic on ports
288 bridge vlan add dev lan1 vid 1 pvid untagged
289 bridge vlan add dev lan2 vid 1 pvid untagged
290 bridge vlan add dev wan vid 2 pvid untagged
292 # configure the VLANs
293 ip addr add 192.0.2.1/30 dev eth0.2
294 ip addr add 192.0.2.129/25 dev br0
296 # bring up the bridge devices
299 Forwarding database (FDB) management
300 ------------------------------------
302 The existing DSA switches do not have the necessary hardware support to keep
303 the software FDB of the bridge in sync with the hardware tables, so the two
304 tables are managed separately (``bridge fdb show`` queries both, and depending
305 on whether the ``self`` or ``master`` flags are being used, a ``bridge fdb
306 add`` or ``bridge fdb del`` command acts upon entries from one or both tables).
308 Up until kernel v4.14, DSA only supported user space management of bridge FDB
309 entries using the bridge bypass operations (which do not update the software
310 FDB, just the hardware one) using the ``self`` flag (which is optional and can
315 bridge fdb add dev swp0 00:01:02:03:04:05 self static
317 bridge fdb add dev swp0 00:01:02:03:04:05 static
319 Due to a bug, the bridge bypass FDB implementation provided by DSA did not
320 distinguish between ``static`` and ``local`` FDB entries (``static`` are meant
321 to be forwarded, while ``local`` are meant to be locally terminated, i.e. sent
322 to the host port). Instead, all FDB entries with the ``self`` flag (implicit or
323 explicit) are treated by DSA as ``static`` even if they are ``local``.
328 bridge fdb add dev swp0 00:01:02:03:04:05 static
329 # behaves the same for DSA as this command:
330 bridge fdb add dev swp0 00:01:02:03:04:05 local
331 # or shorthand, because the 'local' flag is implicit if 'static' is not
332 # specified, it also behaves the same as:
333 bridge fdb add dev swp0 00:01:02:03:04:05
335 The last command is an incorrect way of adding a static bridge FDB entry to a
336 DSA switch using the bridge bypass operations, and works by mistake. Other
337 drivers will treat an FDB entry added by the same command as ``local`` and as
338 such, will not forward it, as opposed to DSA.
340 Between kernel v4.14 and v5.14, DSA has supported in parallel two modes of
341 adding a bridge FDB entry to the switch: the bridge bypass discussed above, as
342 well as a new mode using the ``master`` flag which installs FDB entries in the
347 bridge fdb add dev swp0 00:01:02:03:04:05 master static
349 Since kernel v5.14, DSA has gained stronger integration with the bridge's
350 software FDB, and the support for its bridge bypass FDB implementation (using
351 the ``self`` flag) has been removed. This results in the following changes:
355 # This is the only valid way of adding an FDB entry that is supported,
356 # compatible with v4.14 kernels and later:
357 bridge fdb add dev swp0 00:01:02:03:04:05 master static
358 # This command is no longer buggy and the entry is properly treated as
359 # 'local' instead of being forwarded:
360 bridge fdb add dev swp0 00:01:02:03:04:05
361 # This command no longer installs a static FDB entry to hardware:
362 bridge fdb add dev swp0 00:01:02:03:04:05 static
364 Script writers are therefore encouraged to use the ``master static`` set of
365 flags when working with bridge FDB entries on DSA switch interfaces.
367 Affinity of user ports to CPU ports
368 -----------------------------------
370 Typically, DSA switches are attached to the host via a single Ethernet
371 interface, but in cases where the switch chip is discrete, the hardware design
372 may permit the use of 2 or more ports connected to the host, for an increase in
373 termination throughput.
375 DSA can make use of multiple CPU ports in two ways. First, it is possible to
376 statically assign the termination traffic associated with a certain user port
377 to be processed by a certain CPU port. This way, user space can implement
378 custom policies of static load balancing between user ports, by spreading the
379 affinities according to the available CPU ports.
381 Secondly, it is possible to perform load balancing between CPU ports on a per
382 packet basis, rather than statically assigning user ports to CPU ports.
383 This can be achieved by placing the DSA masters under a LAG interface (bonding
384 or team). DSA monitors this operation and creates a mirror of this software LAG
385 on the CPU ports facing the physical DSA masters that constitute the LAG slave
388 To make use of multiple CPU ports, the firmware (device tree) description of
389 the switch must mark all the links between CPU ports and their DSA masters
390 using the ``ethernet`` reference/phandle. At startup, only a single CPU port
391 and DSA master will be used - the numerically first port from the firmware
392 description which has an ``ethernet`` property. It is up to the user to
393 configure the system for the switch to use other masters.
395 DSA uses the ``rtnl_link_ops`` mechanism (with a "dsa" ``kind``) to allow
396 changing the DSA master of a user port. The ``IFLA_DSA_MASTER`` u32 netlink
397 attribute contains the ifindex of the master device that handles each slave
398 device. The DSA master must be a valid candidate based on firmware node
399 information, or a LAG interface which contains only slaves which are valid
402 Using iproute2, the following manipulations are possible:
406 # See the DSA master in current use
407 ip -d link show dev swp0
411 # Static CPU port distribution
412 ip link set swp0 type dsa master eth1
413 ip link set swp1 type dsa master eth0
414 ip link set swp2 type dsa master eth1
415 ip link set swp3 type dsa master eth0
417 # CPU ports in LAG, using explicit assignment of the DSA master
418 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
419 ip link set eth1 down && ip link set eth1 master bond0
420 ip link set swp0 type dsa master bond0
421 ip link set swp1 type dsa master bond0
422 ip link set swp2 type dsa master bond0
423 ip link set swp3 type dsa master bond0
424 ip link set eth0 down && ip link set eth0 master bond0
425 ip -d link show dev swp0
429 # CPU ports in LAG, relying on implicit migration of the DSA master
430 ip link add bond0 type bond mode balance-xor && ip link set bond0 up
431 ip link set eth0 down && ip link set eth0 master bond0
432 ip link set eth1 down && ip link set eth1 master bond0
433 ip -d link show dev swp0
437 Notice that in the case of CPU ports under a LAG, the use of the
438 ``IFLA_DSA_MASTER`` netlink attribute is not strictly needed, but rather, DSA
439 reacts to the ``IFLA_MASTER`` attribute change of its present master (``eth0``)
440 and migrates all user ports to the new upper of ``eth0``, ``bond0``. Similarly,
441 when ``bond0`` is destroyed using ``RTM_DELLINK``, DSA migrates the user ports
442 that were assigned to this interface to the first physical DSA master which is
443 eligible, based on the firmware description (it effectively reverts to the
444 startup configuration).
446 In a setup with more than 2 physical CPU ports, it is therefore possible to mix
447 static user to CPU port assignment with LAG between DSA masters. It is not
448 possible to statically assign a user port towards a DSA master that has any
449 upper interfaces (this includes LAG devices - the master must always be the LAG
452 Live changing of the DSA master (and thus CPU port) affinity of a user port is
453 permitted, in order to allow dynamic redistribution in response to traffic.
455 Physical DSA masters are allowed to join and leave at any time a LAG interface
456 used as a DSA master; however, DSA will reject a LAG interface as a valid
457 candidate for being a DSA master unless it has at least one physical DSA master