Commit | Line | Data |
---|---|---|
5ee6842c MB |
1 | The QorIQ DPAA Ethernet Driver |
2 | ============================== | |
3 | ||
4 | Authors: | |
5 | Madalin Bucur <madalin.bucur@nxp.com> | |
6 | Camelia Groza <camelia.groza@nxp.com> | |
7 | ||
8 | Contents | |
9 | ======== | |
10 | ||
11 | - DPAA Ethernet Overview | |
12 | - DPAA Ethernet Supported SoCs | |
13 | - Configuring DPAA Ethernet in your kernel | |
14 | - DPAA Ethernet Frame Processing | |
15 | - DPAA Ethernet Features | |
06591916 | 16 | - DPAA IRQ Affinity and Receive Side Scaling |
5ee6842c MB |
17 | - Debugging |
18 | ||
19 | DPAA Ethernet Overview | |
20 | ====================== | |
21 | ||
22 | DPAA stands for Data Path Acceleration Architecture and it is a | |
23 | set of networking acceleration IPs that are available on several | |
24 | generations of SoCs, both on PowerPC and ARM64. | |
25 | ||
26 | The Freescale DPAA architecture consists of a series of hardware blocks | |
27 | that support Ethernet connectivity. The Ethernet driver depends upon the | |
28 | following drivers in the Linux kernel: | |
29 | ||
30 | - Peripheral Access Memory Unit (PAMU) (* needed only for PPC platforms) | |
31 | drivers/iommu/fsl_* | |
32 | - Frame Manager (FMan) | |
33 | drivers/net/ethernet/freescale/fman | |
34 | - Queue Manager (QMan), Buffer Manager (BMan) | |
35 | drivers/soc/fsl/qbman | |
36 | ||
37 | A simplified view of the dpaa_eth interfaces mapped to FMan MACs: | |
38 | ||
39 | dpaa_eth /eth0\ ... /ethN\ | |
40 | driver | | | | | |
41 | ------------- ---- ----------- ---- ------------- | |
42 | -Ports / Tx Rx \ ... / Tx Rx \ | |
43 | FMan | | | | | |
44 | -MACs | MAC0 | | MACN | | |
45 | / dtsec0 \ ... / dtsecN \ (or tgec) | |
46 | / \ / \(or memac) | |
47 | --------- -------------- --- -------------- --------- | |
48 | FMan, FMan Port, FMan SP, FMan MURAM drivers | |
49 | --------------------------------------------------------- | |
50 | FMan HW blocks: MURAM, MACs, Ports, SP | |
51 | --------------------------------------------------------- | |
52 | ||
53 | The dpaa_eth relation to the QMan, BMan and FMan: | |
54 | ________________________________ | |
55 | dpaa_eth / eth0 \ | |
56 | driver / \ | |
57 | --------- -^- -^- -^- --- --------- | |
58 | QMan driver / \ / \ / \ \ / | BMan | | |
59 | |Rx | |Rx | |Tx | |Tx | | driver | | |
60 | --------- |Dfl| |Err| |Cnf| |FQs| | | | |
61 | QMan HW |FQ | |FQ | |FQs| | | | | | |
62 | / \ / \ / \ \ / | | | |
63 | --------- --- --- --- -v- --------- | |
64 | | FMan QMI | | | |
65 | | FMan HW FMan BMI | BMan HW | | |
66 | ----------------------- -------- | |
67 | ||
68 | where the acronyms used above (and in the code) are: | |
69 | DPAA = Data Path Acceleration Architecture | |
70 | FMan = DPAA Frame Manager | |
71 | QMan = DPAA Queue Manager | |
72 | BMan = DPAA Buffers Manager | |
73 | QMI = QMan interface in FMan | |
74 | BMI = BMan interface in FMan | |
75 | FMan SP = FMan Storage Profiles | |
76 | MURAM = Multi-user RAM in FMan | |
77 | FQ = QMan Frame Queue | |
78 | Rx Dfl FQ = default reception FQ | |
79 | Rx Err FQ = Rx error frames FQ | |
80 | Tx Cnf FQ = Tx confirmation FQs | |
81 | Tx FQs = transmission frame queues | |
82 | dtsec = datapath three speed Ethernet controller (10/100/1000 Mbps) | |
83 | tgec = ten gigabit Ethernet controller (10 Gbps) | |
84 | memac = multirate Ethernet MAC (10/100/1000/10000) | |
85 | ||
86 | DPAA Ethernet Supported SoCs | |
87 | ============================ | |
88 | ||
89 | The DPAA drivers enable the Ethernet controllers present on the following SoCs: | |
90 | ||
91 | # PPC | |
92 | P1023 | |
93 | P2041 | |
94 | P3041 | |
95 | P4080 | |
96 | P5020 | |
97 | P5040 | |
98 | T1023 | |
99 | T1024 | |
100 | T1040 | |
101 | T1042 | |
102 | T2080 | |
103 | T4240 | |
104 | B4860 | |
105 | ||
106 | # ARM | |
107 | LS1043A | |
108 | LS1046A | |
109 | ||
110 | Configuring DPAA Ethernet in your kernel | |
111 | ======================================== | |
112 | ||
113 | To enable the DPAA Ethernet driver, the following Kconfig options are required: | |
114 | ||
115 | # common for arch/arm64 and arch/powerpc platforms | |
116 | CONFIG_FSL_DPAA=y | |
117 | CONFIG_FSL_FMAN=y | |
118 | CONFIG_FSL_DPAA_ETH=y | |
119 | CONFIG_FSL_XGMAC_MDIO=y | |
120 | ||
121 | # for arch/powerpc only | |
122 | CONFIG_FSL_PAMU=y | |
123 | ||
124 | # common options needed for the PHYs used on the RDBs | |
125 | CONFIG_VITESSE_PHY=y | |
126 | CONFIG_REALTEK_PHY=y | |
127 | CONFIG_AQUANTIA_PHY=y | |
128 | ||
129 | DPAA Ethernet Frame Processing | |
130 | ============================== | |
131 | ||
132 | On Rx, buffers for the incoming frames are retrieved from one of the three | |
133 | existing buffers pools. The driver initializes and seeds these, each with | |
134 | buffers of different sizes: 1KB, 2KB and 4KB. | |
135 | ||
136 | On Tx, all transmitted frames are returned to the driver through Tx | |
137 | confirmation frame queues. The driver is then responsible for freeing the | |
138 | buffers. In order to do this properly, a backpointer is added to the buffer | |
139 | before transmission that points to the skb. When the buffer returns to the | |
140 | driver on a confirmation FQ, the skb can be correctly consumed. | |
141 | ||
142 | DPAA Ethernet Features | |
143 | ====================== | |
144 | ||
145 | Currently the DPAA Ethernet driver enables the basic features required for | |
146 | a Linux Ethernet driver. The support for advanced features will be added | |
147 | gradually. | |
148 | ||
149 | The driver has Rx and Tx checksum offloading for UDP and TCP. Currently the Rx | |
150 | checksum offload feature is enabled by default and cannot be controlled through | |
06591916 MB |
151 | ethtool. Also, rx-flow-hash and rx-hashing was added. The addition of RSS |
152 | provides a big performance boost for the forwarding scenarios, allowing | |
153 | different traffic flows received by one interface to be processed by different | |
154 | CPUs in parallel. | |
5ee6842c MB |
155 | |
156 | The driver has support for multiple prioritized Tx traffic classes. Priorities | |
157 | range from 0 (lowest) to 3 (highest). These are mapped to HW workqueues with | |
158 | strict priority levels. Each traffic class contains NR_CPU TX queues. By | |
159 | default, only one traffic class is enabled and the lowest priority Tx queues | |
160 | are used. Higher priority traffic classes can be enabled with the mqprio | |
161 | qdisc. For example, all four traffic classes are enabled on an interface with | |
162 | the following command. Furthermore, skb priority levels are mapped to traffic | |
163 | classes as follows: | |
164 | ||
165 | * priorities 0 to 3 - traffic class 0 (low priority) | |
166 | * priorities 4 to 7 - traffic class 1 (medium-low priority) | |
167 | * priorities 8 to 11 - traffic class 2 (medium-high priority) | |
168 | * priorities 12 to 15 - traffic class 3 (high priority) | |
169 | ||
170 | tc qdisc add dev <int> root handle 1: \ | |
171 | mqprio num_tc 4 map 0 0 0 0 1 1 1 1 2 2 2 2 3 3 3 3 hw 1 | |
172 | ||
06591916 MB |
173 | DPAA IRQ Affinity and Receive Side Scaling |
174 | ========================================== | |
175 | ||
176 | Traffic coming on the DPAA Rx queues or on the DPAA Tx confirmation | |
177 | queues is seen by the CPU as ingress traffic on a certain portal. | |
178 | The DPAA QMan portal interrupts are affined each to a certain CPU. | |
179 | The same portal interrupt services all the QMan portal consumers. | |
180 | ||
181 | By default the DPAA Ethernet driver enables RSS, making use of the | |
182 | DPAA FMan Parser and Keygen blocks to distribute traffic on 128 | |
183 | hardware frame queues using a hash on IP v4/v6 source and destination | |
184 | and L4 source and destination ports, in present in the received frame. | |
185 | When RSS is disabled, all traffic received by a certain interface is | |
186 | received on the default Rx frame queue. The default DPAA Rx frame | |
187 | queues are configured to put the received traffic into a pool channel | |
188 | that allows any available CPU portal to dequeue the ingress traffic. | |
189 | The default frame queues have the HOLDACTIVE option set, ensuring that | |
190 | traffic bursts from a certain queue are serviced by the same CPU. | |
191 | This ensures a very low rate of frame reordering. A drawback of this | |
192 | is that only one CPU at a time can service the traffic received by a | |
193 | certain interface when RSS is not enabled. | |
194 | ||
195 | To implement RSS, the DPAA Ethernet driver allocates an extra set of | |
196 | 128 Rx frame queues that are configured to dedicated channels, in a | |
197 | round-robin manner. The mapping of the frame queues to CPUs is now | |
198 | hardcoded, there is no indirection table to move traffic for a certain | |
199 | FQ (hash result) to another CPU. The ingress traffic arriving on one | |
200 | of these frame queues will arrive at the same portal and will always | |
201 | be processed by the same CPU. This ensures intra-flow order preservation | |
202 | and workload distribution for multiple traffic flows. | |
203 | ||
204 | RSS can be turned off for a certain interface using ethtool, i.e. | |
205 | ||
206 | # ethtool -N fm1-mac9 rx-flow-hash tcp4 "" | |
207 | ||
208 | To turn it back on, one needs to set rx-flow-hash for tcp4/6 or udp4/6: | |
209 | ||
210 | # ethtool -N fm1-mac9 rx-flow-hash udp4 sfdn | |
211 | ||
212 | There is no independent control for individual protocols, any command | |
213 | run for one of tcp4|udp4|ah4|esp4|sctp4|tcp6|udp6|ah6|esp6|sctp6 is | |
214 | going to control the rx-flow-hashing for all protocols on that interface. | |
215 | ||
216 | Besides using the FMan Keygen computed hash for spreading traffic on the | |
217 | 128 Rx FQs, the DPAA Ethernet driver also sets the skb hash value when | |
218 | the NETIF_F_RXHASH feature is on (active by default). This can be turned | |
219 | on or off through ethtool, i.e.: | |
220 | ||
221 | # ethtool -K fm1-mac9 rx-hashing off | |
222 | # ethtool -k fm1-mac9 | grep hash | |
223 | receive-hashing: off | |
224 | # ethtool -K fm1-mac9 rx-hashing on | |
225 | Actual changes: | |
226 | receive-hashing: on | |
227 | # ethtool -k fm1-mac9 | grep hash | |
228 | receive-hashing: on | |
229 | ||
230 | Please note that Rx hashing depends upon the rx-flow-hashing being on | |
231 | for that interface - turning off rx-flow-hashing will also disable the | |
232 | rx-hashing (without ethtool reporting it as off as that depends on the | |
233 | NETIF_F_RXHASH feature flag). | |
234 | ||
5ee6842c MB |
235 | Debugging |
236 | ========= | |
237 | ||
238 | The following statistics are exported for each interface through ethtool: | |
239 | ||
240 | - interrupt count per CPU | |
241 | - Rx packets count per CPU | |
242 | - Tx packets count per CPU | |
243 | - Tx confirmed packets count per CPU | |
244 | - Tx S/G frames count per CPU | |
245 | - Tx error count per CPU | |
246 | - Rx error count per CPU | |
247 | - Rx error count per type | |
248 | - congestion related statistics: | |
249 | - congestion status | |
250 | - time spent in congestion | |
251 | - number of time the device entered congestion | |
252 | - dropped packets count per cause | |
253 | ||
254 | The driver also exports the following information in sysfs: | |
255 | ||
256 | - the FQ IDs for each FQ type | |
257 | /sys/devices/platform/dpaa-ethernet.0/net/<int>/fqids | |
258 | ||
259 | - the IDs of the buffer pools in use | |
260 | /sys/devices/platform/dpaa-ethernet.0/net/<int>/bpids |