[linux-block.git] / Documentation / fpga / dfl.txt

===============================================================================
              FPGA Device Feature List (DFL) Framework Overview
-------------------------------------------------------------------------------
                Enno Luebbers <enno.luebbers@intel.com>
                Xiao Guangrong <guangrong.xiao@linux.intel.com>
                Wu Hao <hao.wu@intel.com>

The Device Feature List (DFL) FPGA framework (and drivers according to this
this framework) hides the very details of low layer hardwares and provides
unified interfaces to userspace. Applications could use these interfaces to
configure, enumerate, open and access FPGA accelerators on platforms which
implement the DFL in the device memory. Besides this, the DFL framework
enables system level management functions such as FPGA reconfiguration.


Device Feature List (DFL) Overview
==================================
Device Feature List (DFL) defines a linked list of feature headers within the
device MMIO space to provide an extensible way of adding features. Software can
walk through these predefined data structures to enumerate FPGA features:
FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
as illustrated below:

    Header            Header            Header            Header
 +----------+  +-->+----------+  +-->+----------+  +-->+----------+
 |   Type   |  |   |  Type    |  |   |  Type    |  |   |  Type    |
 |   FIU    |  |   | Private  |  |   | Private  |  |   | Private  |
 +----------+  |   | Feature  |  |   | Feature  |  |   | Feature  |
 | Next_DFH |--+   +----------+  |   +----------+  |   +----------+
 +----------+      | Next_DFH |--+   | Next_DFH |--+   | Next_DFH |--> NULL
 |    ID    |      +----------+      +----------+      +----------+
 +----------+      |    ID    |      |    ID    |      |    ID    |
 | Next_AFU |--+   +----------+      +----------+      +----------+
 +----------+  |   | Feature  |      | Feature  |      | Feature  |
 |  Header  |  |   | Register |      | Register |      | Register |
 | Register |  |   |   Set    |      |   Set    |      |   Set    |
 |   Set    |  |   +----------+      +----------+      +----------+
 +----------+  |      Header
               +-->+----------+
                   |   Type   |
                   |   AFU    |
                   +----------+
                   | Next_DFH |--> NULL
                   +----------+
                   |   GUID   |
                   +----------+
                   |  Header  |
                   | Register |
                   |   Set    |
                   +----------+

FPGA Interface Unit (FIU) represents a standalone functional unit for the
interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
descriptions on FME and Port in later sections).

Accelerated Function Unit (AFU) represents a FPGA programmable region and
always connects to a FIU (e.g. a Port) as its child as illustrated above.

Private Features represent sub features of the FIU and AFU. They could be
various function blocks with different IDs, but all private features which
belong to the same FIU or AFU, must be linked to one list via the Next Device
Feature Header (Next_DFH) pointer.

Each FIU, AFU and Private Feature could implement its own functional registers.
The functional register set for FIU and AFU, is named as Header Register Set,
e.g. FME Header Register Set, and the one for Private Feature, is named as
Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.

This Device Feature List provides a way of linking features together, it's
convenient for software to locate each feature by walking through this list,
and can be implemented in register regions of any FPGA device.


FIU - FME (FPGA Management Engine)
==================================
The FPGA Management Engine performs reconfiguration and other infrastructure
functions. Each FPGA device only has one FME.

User-space applications can acquire exclusive access to the FME using open(),
and release it using close().

The following functions are exposed through ioctls:

 Get driver API version (DFL_FPGA_GET_API_VERSION)
 Check for extensions (DFL_FPGA_CHECK_EXTENSION)
 Program bitstream (DFL_FPGA_FME_PORT_PR)

More functions are exposed through sysfs
(/sys/class/fpga_region/regionX/dfl-fme.n/):

 Read bitstream ID (bitstream_id)
     bitstream_id indicates version of the static FPGA region.

 Read bitstream metadata (bitstream_metadata)
     bitstream_metadata includes detailed information of static FPGA region,
     e.g. synthesis date and seed.

 Read number of ports (ports_num)
     one FPGA device may have more than one port, this sysfs interface indicates
     how many ports the FPGA device has.


FIU - PORT
==========
A port represents the interface between the static FPGA fabric and a partially
reconfigurable region containing an AFU. It controls the communication from SW
to the accelerator and exposes features such as reset and debug. Each FPGA
device may have more than one port, but always one AFU per port.


AFU
===
An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
used for accelerator-specific control registers.

User-space applications can acquire exclusive access to an AFU attached to a
port by using open() on the port device node and release it using close().

The following functions are exposed through ioctls:

 Get driver API version (DFL_FPGA_GET_API_VERSION)
 Check for extensions (DFL_FPGA_CHECK_EXTENSION)
 Get port info (DFL_FPGA_PORT_GET_INFO)
 Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
 Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
 Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
 Reset AFU (*DFL_FPGA_PORT_RESET)

*DFL_FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port
reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
never cause any system level issue, only functional failure (e.g. DMA or PR
operation failure) and be recoverable from the failure.

User-space applications can also mmap() accelerator MMIO regions.

More functions are exposed through sysfs:
(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):

 Read Accelerator GUID (afu_id)
     afu_id indicates which PR bitstream is programmed to this AFU.


DFL Framework Overview
======================

         +----------+    +--------+ +--------+ +--------+
         |   FME    |    |  AFU   | |  AFU   | |  AFU   |
         |  Module  |    | Module | | Module | | Module |
         +----------+    +--------+ +--------+ +--------+
                 +-----------------------+
                 | FPGA Container Device |    Device Feature List
                 |  (FPGA Base Region)   |         Framework
                 +-----------------------+
--------------------------------------------------------------------
               +----------------------------+
               |   FPGA DFL Device Module   |
               | (e.g. PCIE/Platform Device)|
               +----------------------------+
                 +------------------------+
                 |  FPGA Hardware Device  |
                 +------------------------+

DFL framework in kernel provides common interfaces to create container device
(FPGA base region), discover feature devices and their private features from the
given Device Feature Lists and create platform devices for feature devices
(e.g. FME, Port and AFU) with related resources under the container device. It
also abstracts operations for the private features and exposes common ops to
feature device drivers.

The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
device and etc. Its driver module is always loaded first once the device is
created by the system. This driver plays an infrastructural role in the
driver architecture. It locates the DFLs in the device memory, handles them
and related resources to common interfaces from DFL framework for enumeration.
(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).

The FPGA Management Engine (FME) driver is a platform driver which is loaded
automatically after FME platform device creation from the DFL device module. It
provides the key features for FPGA management, including:

	a) Expose static FPGA region information, e.g. version and metadata.
	   Users can read related information via sysfs interfaces exposed
	   by FME driver.

	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
	   bridges and FPGA regions during PR sub feature initialization. Once
	   it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
	   common interface function from FPGA Region to complete the partial
	   reconfiguration of the PR bitstream to the given port.

Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
probed once the AFU platform device is created. The main function of this module
is to provide an interface for userspace applications to access the individual
accelerators, including basic reset control on port, AFU MMIO region export, dma
buffer mapping service functions.

After feature platform devices creation, matched platform drivers will be loaded
automatically to handle different functionalities. Please refer to next sections
for detailed information on functional units which have been already implemented
under this DFL framework.


Partial Reconfiguration
=======================
As mentioned above, accelerators can be reconfigured through partial
reconfiguration of a PR bitstream file. The PR bitstream file must have been
generated for the exact static FPGA region and targeted reconfigurable region
(port) of the FPGA, otherwise, the reconfiguration operation will fail and
possibly cause system instability. This compatibility can be checked by
comparing the compatibility ID noted in the header of PR bitstream file against
the compat_id exposed by the target FPGA region. This check is usually done by
userspace before calling the reconfiguration IOCTL.


Device enumeration
==================
This section introduces how applications enumerate the fpga device from
the sysfs hierarchy under /sys/class/fpga_region.

In the example below, two DFL based FPGA devices are installed in the host. Each
fpga device has one FME and two ports (AFUs).

FPGA regions are created under /sys/class/fpga_region/

	/sys/class/fpga_region/region0
	/sys/class/fpga_region/region1
	/sys/class/fpga_region/region2
	...

Application needs to search each regionX folder, if feature device is found,
(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
fpga region which represents the FPGA device.

Each base region has one FME and two ports (AFUs) as child devices:

	/sys/class/fpga_region/region0/dfl-fme.0
	/sys/class/fpga_region/region0/dfl-port.0
	/sys/class/fpga_region/region0/dfl-port.1
	...

	/sys/class/fpga_region/region3/dfl-fme.1
	/sys/class/fpga_region/region3/dfl-port.2
	/sys/class/fpga_region/region3/dfl-port.3
	...

In general, the FME/AFU sysfs interfaces are named as follows:

	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
	/sys/class/fpga_region/<regionX>/<dfl-port.m>/

with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
ports.

The device nodes used for ioctl() or mmap() can be referenced through:

	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev


Add new FIUs support
====================
It's possible that developers made some new function blocks (FIUs) under this
DFL framework, then new platform device driver needs to be developed for the
new feature dev (FIU) following the same way as existing feature dev drivers
(e.g. FME and Port/AFU platform device driver). Besides that, it requires
modification on DFL framework enumeration code too, for new FIU type detection
and related platform devices creation.


Add new private features support
================================
In some cases, we may need to add some new private features to existing FIUs
(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
framework, as each private feature will be parsed automatically and related
mmio resources can be found under FIU platform device created by DFL framework.
Developer only needs to provide a sub feature driver with matched feature id.
FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
could be a reference.


Open discussion
===============
FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
to user now. In the future, if unified user interfaces for reconfiguration are
added, FME driver should switch to them from ioctl interface.
Commit	Line	Data
c73c9ad2 WH	1	===============================================================================
	2	FPGA Device Feature List (DFL) Framework Overview
	3	-------------------------------------------------------------------------------
	4	Enno Luebbers <enno.luebbers@intel.com>
	5	Xiao Guangrong <guangrong.xiao@linux.intel.com>
	6	Wu Hao <hao.wu@intel.com>
	7
	8	The Device Feature List (DFL) FPGA framework (and drivers according to this
	9	this framework) hides the very details of low layer hardwares and provides
	10	unified interfaces to userspace. Applications could use these interfaces to
	11	configure, enumerate, open and access FPGA accelerators on platforms which
	12	implement the DFL in the device memory. Besides this, the DFL framework
	13	enables system level management functions such as FPGA reconfiguration.
	14
	15
	16	Device Feature List (DFL) Overview
	17	==================================
	18	Device Feature List (DFL) defines a linked list of feature headers within the
	19	device MMIO space to provide an extensible way of adding features. Software can
	20	walk through these predefined data structures to enumerate FPGA features:
	21	FPGA Interface Unit (FIU), Accelerated Function Unit (AFU) and Private Features,
	22	as illustrated below:
	23
	24	Header Header Header Header
	25	+----------+ +-->+----------+ +-->+----------+ +-->+----------+
	26	\| Type \| \| \| Type \| \| \| Type \| \| \| Type \|
	27	\| FIU \| \| \| Private \| \| \| Private \| \| \| Private \|
	28	+----------+ \| \| Feature \| \| \| Feature \| \| \| Feature \|
	29	\| Next_DFH \|--+ +----------+ \| +----------+ \| +----------+
	30	+----------+ \| Next_DFH \|--+ \| Next_DFH \|--+ \| Next_DFH \|--> NULL
	31	\| ID \| +----------+ +----------+ +----------+
	32	+----------+ \| ID \| \| ID \| \| ID \|
	33	\| Next_AFU \|--+ +----------+ +----------+ +----------+
	34	+----------+ \| \| Feature \| \| Feature \| \| Feature \|
	35	\| Header \| \| \| Register \| \| Register \| \| Register \|
	36	\| Register \| \| \| Set \| \| Set \| \| Set \|
	37	\| Set \| \| +----------+ +----------+ +----------+
	38	+----------+ \| Header
	39	+-->+----------+
	40	\| Type \|
	41	\| AFU \|
	42	+----------+
	43	\| Next_DFH \|--> NULL
	44	+----------+
	45	\| GUID \|
	46	+----------+
	47	\| Header \|
	48	\| Register \|
	49	\| Set \|
	50	+----------+
	51
	52	FPGA Interface Unit (FIU) represents a standalone functional unit for the
	53	interface to FPGA, e.g. the FPGA Management Engine (FME) and Port (more
	54	descriptions on FME and Port in later sections).
	55
	56	Accelerated Function Unit (AFU) represents a FPGA programmable region and
	57	always connects to a FIU (e.g. a Port) as its child as illustrated above.
	58
	59	Private Features represent sub features of the FIU and AFU. They could be
	60	various function blocks with different IDs, but all private features which
	61	belong to the same FIU or AFU, must be linked to one list via the Next Device
	62	Feature Header (Next_DFH) pointer.
	63
	64	Each FIU, AFU and Private Feature could implement its own functional registers.
65	The functional register set for FIU and AFU, is named as Header Register Set,
66	e.g. FME Header Register Set, and the one for Private Feature, is named as
67	Feature Register Set, e.g. FME Partial Reconfiguration Feature Register Set.
68
69	This Device Feature List provides a way of linking features together, it's
70	convenient for software to locate each feature by walking through this list,
71	and can be implemented in register regions of any FPGA device.
72
73
74	FIU - FME (FPGA Management Engine)
75	==================================
76	The FPGA Management Engine performs reconfiguration and other infrastructure
77	functions. Each FPGA device only has one FME.
78
79	User-space applications can acquire exclusive access to the FME using open(),
80	and release it using close().
81
82	The following functions are exposed through ioctls:
83
84	Get driver API version (DFL_FPGA_GET_API_VERSION)
85	Check for extensions (DFL_FPGA_CHECK_EXTENSION)
86	Program bitstream (DFL_FPGA_FME_PORT_PR)
87
88	More functions are exposed through sysfs
89	(/sys/class/fpga_region/regionX/dfl-fme.n/):
90
91	Read bitstream ID (bitstream_id)
92	bitstream_id indicates version of the static FPGA region.
93
94	Read bitstream metadata (bitstream_metadata)
95	bitstream_metadata includes detailed information of static FPGA region,
96	e.g. synthesis date and seed.
97
98	Read number of ports (ports_num)
99	one FPGA device may have more than one port, this sysfs interface indicates
100	how many ports the FPGA device has.
101
102
103	FIU - PORT
104	==========
105	A port represents the interface between the static FPGA fabric and a partially
106	reconfigurable region containing an AFU. It controls the communication from SW
107	to the accelerator and exposes features such as reset and debug. Each FPGA
108	device may have more than one port, but always one AFU per port.
109
110
111	AFU
112	===
113	An AFU is attached to a port FIU and exposes a fixed length MMIO region to be
114	used for accelerator-specific control registers.
115
116	User-space applications can acquire exclusive access to an AFU attached to a
117	port by using open() on the port device node and release it using close().
118
119	The following functions are exposed through ioctls:
120
121	Get driver API version (DFL_FPGA_GET_API_VERSION)
122	Check for extensions (DFL_FPGA_CHECK_EXTENSION)
123	Get port info (DFL_FPGA_PORT_GET_INFO)
124	Get MMIO region info (DFL_FPGA_PORT_GET_REGION_INFO)
125	Map DMA buffer (DFL_FPGA_PORT_DMA_MAP)
126	Unmap DMA buffer (DFL_FPGA_PORT_DMA_UNMAP)
127	Reset AFU (*DFL_FPGA_PORT_RESET)
128
129	*DFL_FPGA_PORT_RESET: reset the FPGA Port and its AFU. Userspace can do Port
130	reset at any time, e.g. during DMA or Partial Reconfiguration. But it should
131	never cause any system level issue, only functional failure (e.g. DMA or PR
132	operation failure) and be recoverable from the failure.
133
134	User-space applications can also mmap() accelerator MMIO regions.
135
136	More functions are exposed through sysfs:
137	(/sys/class/fpga_region/<regionX>/<dfl-port.m>/):
138
139	Read Accelerator GUID (afu_id)
140	afu_id indicates which PR bitstream is programmed to this AFU.
141
142
143	DFL Framework Overview
144	======================
145
146	+----------+ +--------+ +--------+ +--------+
147	\| FME \| \| AFU \| \| AFU \| \| AFU \|
148	\| Module \| \| Module \| \| Module \| \| Module \|
149	+----------+ +--------+ +--------+ +--------+
150	+-----------------------+
151	\| FPGA Container Device \| Device Feature List
152	\| (FPGA Base Region) \| Framework
153	+-----------------------+
154	--------------------------------------------------------------------
155	+----------------------------+
156	\| FPGA DFL Device Module \|
157	\| (e.g. PCIE/Platform Device)\|
158	+----------------------------+
159	+------------------------+
160	\| FPGA Hardware Device \|
161	+------------------------+
162
163	DFL framework in kernel provides common interfaces to create container device
164	(FPGA base region), discover feature devices and their private features from the
165	given Device Feature Lists and create platform devices for feature devices
166	(e.g. FME, Port and AFU) with related resources under the container device. It
167	also abstracts operations for the private features and exposes common ops to
168	feature device drivers.
169
170	The FPGA DFL Device could be different hardwares, e.g. PCIe device, platform
171	device and etc. Its driver module is always loaded first once the device is
172	created by the system. This driver plays an infrastructural role in the
173	driver architecture. It locates the DFLs in the device memory, handles them
174	and related resources to common interfaces from DFL framework for enumeration.
175	(Please refer to drivers/fpga/dfl.c for detailed enumeration APIs).
176
177	The FPGA Management Engine (FME) driver is a platform driver which is loaded
178	automatically after FME platform device creation from the DFL device module. It
179	provides the key features for FPGA management, including:
180
181	a) Expose static FPGA region information, e.g. version and metadata.
182	Users can read related information via sysfs interfaces exposed
183	by FME driver.
184
185	b) Partial Reconfiguration. The FME driver creates FPGA manager, FPGA
186	bridges and FPGA regions during PR sub feature initialization. Once
187	it receives a DFL_FPGA_FME_PORT_PR ioctl from user, it invokes the
188	common interface function from FPGA Region to complete the partial
189	reconfiguration of the PR bitstream to the given port.
190
191	Similar to the FME driver, the FPGA Accelerated Function Unit (AFU) driver is
192	probed once the AFU platform device is created. The main function of this module
193	is to provide an interface for userspace applications to access the individual
194	accelerators, including basic reset control on port, AFU MMIO region export, dma
195	buffer mapping service functions.
196
197	After feature platform devices creation, matched platform drivers will be loaded
198	automatically to handle different functionalities. Please refer to next sections
199	for detailed information on functional units which have been already implemented
200	under this DFL framework.
201
202
203	Partial Reconfiguration
204	=======================
205	As mentioned above, accelerators can be reconfigured through partial
206	reconfiguration of a PR bitstream file. The PR bitstream file must have been
207	generated for the exact static FPGA region and targeted reconfigurable region
208	(port) of the FPGA, otherwise, the reconfiguration operation will fail and
209	possibly cause system instability. This compatibility can be checked by
210	comparing the compatibility ID noted in the header of PR bitstream file against
211	the compat_id exposed by the target FPGA region. This check is usually done by
212	userspace before calling the reconfiguration IOCTL.
213
214
215	Device enumeration
216	==================
217	This section introduces how applications enumerate the fpga device from
218	the sysfs hierarchy under /sys/class/fpga_region.
219
220	In the example below, two DFL based FPGA devices are installed in the host. Each
221	fpga device has one FME and two ports (AFUs).
222
223	FPGA regions are created under /sys/class/fpga_region/
224
225	/sys/class/fpga_region/region0
226	/sys/class/fpga_region/region1
227	/sys/class/fpga_region/region2
228	...
229
230	Application needs to search each regionX folder, if feature device is found,
231	(e.g. "dfl-port.n" or "dfl-fme.m" is found), then it's the base
232	fpga region which represents the FPGA device.
233
234	Each base region has one FME and two ports (AFUs) as child devices:
235
236	/sys/class/fpga_region/region0/dfl-fme.0
237	/sys/class/fpga_region/region0/dfl-port.0
238	/sys/class/fpga_region/region0/dfl-port.1
239	...
240
241	/sys/class/fpga_region/region3/dfl-fme.1
242	/sys/class/fpga_region/region3/dfl-port.2
243	/sys/class/fpga_region/region3/dfl-port.3
244	...
245
246	In general, the FME/AFU sysfs interfaces are named as follows:
247
248	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/
249	/sys/class/fpga_region/<regionX>/<dfl-port.m>/
250
251	with 'n' consecutively numbering all FMEs and 'm' consecutively numbering all
252	ports.
253
254	The device nodes used for ioctl() or mmap() can be referenced through:
255
256	/sys/class/fpga_region/<regionX>/<dfl-fme.n>/dev
257	/sys/class/fpga_region/<regionX>/<dfl-port.n>/dev
258
259
260	Add new FIUs support
261	====================
262	It's possible that developers made some new function blocks (FIUs) under this
263	DFL framework, then new platform device driver needs to be developed for the
264	new feature dev (FIU) following the same way as existing feature dev drivers
265	(e.g. FME and Port/AFU platform device driver). Besides that, it requires
266	modification on DFL framework enumeration code too, for new FIU type detection
267	and related platform devices creation.
268
269
270	Add new private features support
271	================================
272	In some cases, we may need to add some new private features to existing FIUs
273	(e.g. FME or Port). Developers don't need to touch enumeration code in DFL
274	framework, as each private feature will be parsed automatically and related
275	mmio resources can be found under FIU platform device created by DFL framework.
276	Developer only needs to provide a sub feature driver with matched feature id.
277	FME Partial Reconfiguration Sub Feature driver (see drivers/fpga/dfl-fme-pr.c)
278	could be a reference.
279
280
281	Open discussion
282	===============
283	FME driver exports one ioctl (DFL_FPGA_FME_PORT_PR) for partial reconfiguration
284	to user now. In the future, if unified user interfaces for reconfiguration are
285	added, FME driver should switch to them from ioctl interface.