Commit | Line | Data |
---|---|---|
c9eda125 SY |
1 | Copyright (C) 2015 Freescale Semiconductor Inc. |
2 | ||
3 | DPAA2 (Data Path Acceleration Architecture Gen2) | |
4 | ------------------------------------------------ | |
5 | ||
6 | This document provides an overview of the Freescale DPAA2 architecture | |
7 | and how it is integrated into the Linux kernel. | |
8 | ||
9 | Contents summary | |
10 | -DPAA2 overview | |
11 | -Overview of DPAA2 objects | |
12 | -DPAA2 Linux driver architecture overview | |
13 | -bus driver | |
14 | -dprc driver | |
15 | -allocator | |
16 | -dpio driver | |
17 | -Ethernet | |
18 | -mac | |
19 | ||
20 | DPAA2 Overview | |
21 | -------------- | |
22 | ||
23 | DPAA2 is a hardware architecture designed for high-speeed network | |
24 | packet processing. DPAA2 consists of sophisticated mechanisms for | |
25 | processing Ethernet packets, queue management, buffer management, | |
26 | autonomous L2 switching, virtual Ethernet bridging, and accelerator | |
27 | (e.g. crypto) sharing. | |
28 | ||
29 | A DPAA2 hardware component called the Management Complex (or MC) manages the | |
30 | DPAA2 hardware resources. The MC provides an object-based abstraction for | |
31 | software drivers to use the DPAA2 hardware. | |
32 | ||
33 | The MC uses DPAA2 hardware resources such as queues, buffer pools, and | |
34 | network ports to create functional objects/devices such as network | |
35 | interfaces, an L2 switch, or accelerator instances. | |
36 | ||
37 | The MC provides memory-mapped I/O command interfaces (MC portals) | |
38 | which DPAA2 software drivers use to operate on DPAA2 objects: | |
39 | ||
40 | +--------------------------------------+ | |
41 | | OS | | |
42 | | DPAA2 drivers | | |
43 | | | | | |
44 | +-----------------------------|--------+ | |
45 | | | |
46 | | (create,discover,connect | |
47 | | config,use,destroy) | |
48 | | | |
49 | DPAA2 | | |
50 | +------------------------| mc portal |-+ | |
51 | | | | | |
52 | | +- - - - - - - - - - - - -V- - -+ | | |
53 | | | | | | |
54 | | | Management Complex (MC) | | | |
55 | | | | | | |
56 | | +- - - - - - - - - - - - - - - -+ | | |
57 | | | | |
58 | | Hardware Hardware | | |
59 | | Resources Objects | | |
60 | | --------- ------- | | |
61 | | -queues -DPRC | | |
62 | | -buffer pools -DPMCP | | |
63 | | -Eth MACs/ports -DPIO | | |
64 | | -network interface -DPNI | | |
65 | | profiles -DPMAC | | |
66 | | -queue portals -DPBP | | |
67 | | -MC portals ... | | |
68 | | ... | | |
69 | | | | |
70 | +--------------------------------------+ | |
71 | ||
72 | The MC mediates operations such as create, discover, | |
73 | connect, configuration, and destroy. Fast-path operations | |
74 | on data, such as packet transmit/receive, are not mediated by | |
75 | the MC and are done directly using memory mapped regions in | |
76 | DPIO objects. | |
77 | ||
78 | Overview of DPAA2 Objects | |
79 | ------------------------- | |
80 | The section provides a brief overview of some key objects | |
81 | in the DPAA2 hardware. A simple scenario is described illustrating | |
82 | the objects involved in creating a network interfaces. | |
83 | ||
84 | -DPRC (Datapath Resource Container) | |
85 | ||
86 | A DPRC is an container object that holds all the other | |
87 | types of DPAA2 objects. In the example diagram below there | |
88 | are 8 objects of 5 types (DPMCP, DPIO, DPBP, DPNI, and DPMAC) | |
89 | in the container. | |
90 | ||
91 | +---------------------------------------------------------+ | |
92 | | DPRC | | |
93 | | | | |
94 | | +-------+ +-------+ +-------+ +-------+ +-------+ | | |
95 | | | DPMCP | | DPIO | | DPBP | | DPNI | | DPMAC | | | |
96 | | +-------+ +-------+ +-------+ +---+---+ +---+---+ | | |
97 | | | DPMCP | | DPIO | | | |
98 | | +-------+ +-------+ | | |
99 | | | DPMCP | | | |
100 | | +-------+ | | |
101 | | | | |
102 | +---------------------------------------------------------+ | |
103 | ||
104 | From the point of view of an OS, a DPRC is bus-like. Like | |
105 | a plug-and-play bus, such as PCI, DPRC commands can be used to | |
106 | enumerate the contents of the DPRC, discover the hardware | |
107 | objects present (including mappable regions and interrupts). | |
108 | ||
109 | dprc.1 (bus) | |
110 | | | |
111 | +--+--------+-------+-------+-------+ | |
112 | | | | | | | |
113 | dpmcp.1 dpio.1 dpbp.1 dpni.1 dpmac.1 | |
114 | dpmcp.2 dpio.2 | |
115 | dpmcp.3 | |
116 | ||
117 | Hardware objects can be created and destroyed dynamically, providing | |
118 | the ability to hot plug/unplug objects in and out of the DPRC. | |
119 | ||
120 | A DPRC has a mappable mmio region (an MC portal) that can be used | |
121 | to send MC commands. It has an interrupt for status events (like | |
122 | hotplug). | |
123 | ||
124 | All objects in a container share the same hardware "isolation context". | |
125 | This means that with respect to an IOMMU the isolation granularity | |
126 | is at the DPRC (container) level, not at the individual object | |
127 | level. | |
128 | ||
129 | DPRCs can be defined statically and populated with objects | |
130 | via a config file passed to the MC when firmware starts | |
131 | it. There is also a Linux user space tool called "restool" | |
132 | that can be used to create/destroy containers and objects | |
133 | dynamically. | |
134 | ||
135 | -DPAA2 Objects for an Ethernet Network Interface | |
136 | ||
137 | A typical Ethernet NIC is monolithic-- the NIC device contains TX/RX | |
138 | queuing mechanisms, configuration mechanisms, buffer management, | |
139 | physical ports, and interrupts. DPAA2 uses a more granular approach | |
140 | utilizing multiple hardware objects. Each object has specialized | |
141 | functions, and are used together by software to provide Ethernet network | |
142 | interface functionality. This approach provides efficient use of finite | |
143 | hardware resources, flexibility, and performance advantages. | |
144 | ||
145 | The diagram below shows the objects needed for a simple | |
146 | network interface configuration on a system with 2 CPUs. | |
147 | ||
148 | +---+---+ +---+---+ | |
149 | CPU0 CPU1 | |
150 | +---+---+ +---+---+ | |
151 | | | | |
152 | +---+---+ +---+---+ | |
153 | DPIO DPIO | |
154 | +---+---+ +---+---+ | |
155 | \ / | |
156 | \ / | |
157 | \ / | |
158 | +---+---+ | |
159 | DPNI --- DPBP,DPMCP | |
160 | +---+---+ | |
161 | | | |
162 | | | |
163 | +---+---+ | |
164 | DPMAC | |
165 | +---+---+ | |
166 | | | |
167 | port/PHY | |
168 | ||
169 | Below the objects are described. For each object a brief description | |
170 | is provided along with a summary of the kinds of operations the object | |
171 | supports and a summary of key resources of the object (mmio regions | |
172 | and irqs). | |
173 | ||
174 | -DPMAC (Datapath Ethernet MAC): represents an Ethernet MAC, a | |
175 | hardware device that connects to an Ethernet PHY and allows | |
176 | physical transmission and reception of Ethernet frames. | |
177 | -mmio regions: none | |
178 | -irqs: dpni link change | |
179 | -commands: set link up/down, link config, get stats, | |
180 | irq config, enable, reset | |
181 | ||
182 | -DPNI (Datapath Network Interface): contains TX/RX queues, | |
183 | network interface configuration, and rx buffer pool configuration | |
184 | mechanisms. | |
185 | -mmio regions: none | |
186 | -irqs: link state | |
187 | -commands: port config, offload config, queue config, | |
188 | parse/classify config, irq config, enable, reset | |
189 | ||
190 | -DPIO (Datapath I/O): provides interfaces to enqueue and dequeue | |
191 | packets and do hardware buffer pool management operations. For | |
192 | optimum performance there is typically DPIO per CPU. This allows | |
193 | each CPU to perform simultaneous enqueue/dequeue operations. | |
194 | -mmio regions: queue operations, buffer mgmt | |
195 | -irqs: data availability, congestion notification, buffer | |
196 | pool depletion | |
197 | -commands: irq config, enable, reset | |
198 | ||
199 | -DPBP (Datapath Buffer Pool): represents a hardware buffer | |
200 | pool. | |
201 | -mmio regions: none | |
202 | -irqs: none | |
203 | -commands: enable, reset | |
204 | ||
205 | -DPMCP (Datapath MC Portal): provides an MC command portal. | |
206 | Used by drivers to send commands to the MC to manage | |
207 | objects. | |
208 | -mmio regions: MC command portal | |
209 | -irqs: command completion | |
210 | -commands: irq config, enable, reset | |
211 | ||
212 | Object Connections | |
213 | ------------------ | |
214 | Some objects have explicit relationships that must | |
215 | be configured: | |
216 | ||
217 | -DPNI <--> DPMAC | |
218 | -DPNI <--> DPNI | |
219 | -DPNI <--> L2-switch-port | |
220 | A DPNI must be connected to something such as a DPMAC, | |
221 | another DPNI, or L2 switch port. The DPNI connection | |
222 | is made via a DPRC command. | |
223 | ||
224 | +-------+ +-------+ | |
225 | | DPNI | | DPMAC | | |
226 | +---+---+ +---+---+ | |
227 | | | | |
228 | +==========+ | |
229 | ||
230 | -DPNI <--> DPBP | |
231 | A network interface requires a 'buffer pool' (DPBP | |
232 | object) which provides a list of pointers to memory | |
233 | where received Ethernet data is to be copied. The | |
234 | Ethernet driver configures the DPBPs associated with | |
235 | the network interface. | |
236 | ||
237 | Interrupts | |
238 | ---------- | |
239 | All interrupts generated by DPAA2 objects are message | |
240 | interrupts. At the hardware level message interrupts | |
241 | generated by devices will normally have 3 components-- | |
242 | 1) a non-spoofable 'device-id' expressed on the hardware | |
243 | bus, 2) an address, 3) a data value. | |
244 | ||
245 | In the case of DPAA2 devices/objects, all objects in the | |
246 | same container/DPRC share the same 'device-id'. | |
247 | For ARM-based SoC this is the same as the stream ID. | |
248 | ||
249 | ||
250 | DPAA2 Linux Driver Overview | |
251 | --------------------------- | |
252 | ||
253 | This section provides an overview of the Linux kernel drivers for | |
254 | DPAA2-- 1) the bus driver and associated "DPAA2 infrastructure" | |
255 | drivers and 2) functional object drivers (such as Ethernet). | |
256 | ||
257 | As described previously, a DPRC is a container that holds the other | |
258 | types of DPAA2 objects. It is functionally similar to a plug-and-play | |
259 | bus controller. | |
260 | ||
261 | Each object in the DPRC is a Linux "device" and is bound to a driver. | |
262 | The diagram below shows the Linux drivers involved in a networking | |
263 | scenario and the objects bound to each driver. A brief description | |
264 | of each driver follows. | |
265 | ||
266 | +------------+ | |
267 | | OS Network | | |
268 | | Stack | | |
269 | +------------+ +------------+ | |
270 | | Allocator |. . . . . . . | Ethernet | | |
271 | |(dpmcp,dpbp)| | (dpni) | | |
272 | +-.----------+ +---+---+----+ | |
273 | . . ^ | | |
274 | . . <data avail, | |<enqueue, | |
275 | . . tx confirm> | | dequeue> | |
276 | +-------------+ . | | | |
277 | | DPRC driver | . +---+---V----+ +---------+ | |
278 | | (dprc) | . . . . . .| DPIO driver| | MAC | | |
279 | +----------+--+ | (dpio) | | (dpmac) | | |
280 | | +------+-----+ +-----+---+ | |
281 | |<dev add/remove> | | | |
282 | | | | | |
283 | +----+--------------+ | +--+---+ | |
284 | | mc-bus driver | | | PHY | | |
285 | | | | |driver| | |
286 | | /fsl-mc@80c000000 | | +--+---+ | |
287 | +-------------------+ | | | |
288 | | | | |
289 | ================================ HARDWARE =========|=================|====== | |
290 | DPIO | | |
291 | | | | |
292 | DPNI---DPBP | | |
293 | | | | |
294 | DPMAC | | |
295 | | | | |
296 | PHY ---------------+ | |
297 | ===================================================|======================== | |
298 | ||
299 | A brief description of each driver is provided below. | |
300 | ||
301 | mc-bus driver | |
302 | ------------- | |
303 | The mc-bus driver is a platform driver and is probed from an | |
304 | "/fsl-mc@xxxx" node in the device tree passed in by boot firmware. | |
305 | It is responsible for bootstrapping the DPAA2 kernel infrastructure. | |
306 | Key functions include: | |
307 | -registering a new bus type named "fsl-mc" with the kernel, | |
308 | and implementing bus call-backs (e.g. match/uevent/dev_groups) | |
309 | -implemeting APIs for DPAA2 driver registration and for device | |
310 | add/remove | |
311 | -creates an MSI irq domain | |
312 | -do a device add of the 'root' DPRC device, which is needed | |
313 | to bootstrap things | |
314 | ||
315 | DPRC driver | |
316 | ----------- | |
317 | The dprc-driver is bound DPRC objects and does runtime management | |
318 | of a bus instance. It performs the initial bus scan of the DPRC | |
319 | and handles interrupts for container events such as hot plug. | |
320 | ||
321 | Allocator | |
322 | ---------- | |
323 | Certain objects such as DPMCP and DPBP are generic and fungible, | |
324 | and are intended to be used by other drivers. For example, | |
325 | the DPAA2 Ethernet driver needs: | |
326 | -DPMCPs to send MC commands, to configure network interfaces | |
327 | -DPBPs for network buffer pools | |
328 | ||
329 | The allocator driver registers for these allocatable object types | |
330 | and those objects are bound to the allocator when the bus is probed. | |
331 | The allocator maintains a pool of objects that are available for | |
332 | allocation by other DPAA2 drivers. | |
333 | ||
334 | DPIO driver | |
335 | ----------- | |
336 | The DPIO driver is bound to DPIO objects and provides services that allow | |
337 | other drivers such as the Ethernet driver to receive and transmit data. | |
338 | Key services include: | |
339 | -data availability notifications | |
340 | -hardware queuing operations (enqueue and dequeue of data) | |
341 | -hardware buffer pool management | |
342 | ||
343 | There is typically one DPIO object per physical CPU for optimum | |
344 | performance, allowing each CPU to simultaneously enqueue | |
345 | and dequeue data. | |
346 | ||
347 | The DPIO driver operates on behalf of all DPAA2 drivers | |
348 | active in the kernel-- Ethernet, crypto, compression, | |
349 | etc. | |
350 | ||
351 | Ethernet | |
352 | -------- | |
353 | The Ethernet driver is bound to a DPNI and implements the kernel | |
354 | interfaces needed to connect the DPAA2 network interface to | |
355 | the network stack. | |
356 | ||
357 | Each DPNI corresponds to a Linux network interface. | |
358 | ||
359 | MAC driver | |
360 | ---------- | |
361 | An Ethernet PHY is an off-chip, board specific component and is managed | |
362 | by the appropriate PHY driver via an mdio bus. The MAC driver | |
363 | plays a role of being a proxy between the PHY driver and the | |
364 | MC. It does this proxy via the MC commands to a DPMAC object. |