Commit | Line | Data |
---|---|---|
f2ac8ce8 MCC |
1 | .. SPDX-License-Identifier: GPL-2.0 |
2 | ||
684ffa2d MCC |
3 | Media Controller devices |
4 | ------------------------ | |
5 | ||
6 | Media Controller | |
7 | ~~~~~~~~~~~~~~~~ | |
8 | ||
74604b73 | 9 | The media controller userspace API is documented in |
da83c888 | 10 | :ref:`the Media Controller uAPI book <media_controller>`. This document focus |
684ffa2d MCC |
11 | on the kernel-side implementation of the media framework. |
12 | ||
13 | Abstract media device model | |
14 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
15 | ||
16 | Discovering a device internal topology, and configuring it at runtime, is one | |
17 | of the goals of the media framework. To achieve this, hardware devices are | |
18 | modelled as an oriented graph of building blocks called entities connected | |
19 | through pads. | |
20 | ||
21 | An entity is a basic media hardware building block. It can correspond to | |
22 | a large variety of logical blocks such as physical hardware devices | |
23 | (CMOS sensor for instance), logical hardware devices (a building block | |
24 | in a System-on-Chip image processing pipeline), DMA channels or physical | |
25 | connectors. | |
26 | ||
27 | A pad is a connection endpoint through which an entity can interact with | |
28 | other entities. Data (not restricted to video) produced by an entity | |
29 | flows from the entity's output to one or more entity inputs. Pads should | |
30 | not be confused with physical pins at chip boundaries. | |
31 | ||
32 | A link is a point-to-point oriented connection between two pads, either | |
33 | on the same entity or on different entities. Data flows from a source | |
34 | pad to a sink pad. | |
35 | ||
36 | Media device | |
37 | ^^^^^^^^^^^^ | |
38 | ||
9303c9d5 | 39 | A media device is represented by a struct media_device |
74604b73 MCC |
40 | instance, defined in ``include/media/media-device.h``. |
41 | Allocation of the structure is handled by the media device driver, usually by | |
42 | embedding the :c:type:`media_device` instance in a larger driver-specific | |
43 | structure. | |
684ffa2d | 44 | |
1d1d8669 SA |
45 | Drivers initialise media device instances by calling |
46 | :c:func:`media_device_init()`. After initialising a media device instance, it is | |
47 | registered by calling :c:func:`__media_device_register()` via the macro | |
48 | ``media_device_register()`` and unregistered by calling | |
49 | :c:func:`media_device_unregister()`. An initialised media device must be | |
50 | eventually cleaned up by calling :c:func:`media_device_cleanup()`. | |
51 | ||
52 | Note that it is not allowed to unregister a media device instance that was not | |
53 | previously registered, or clean up a media device instance that was not | |
54 | previously initialised. | |
684ffa2d MCC |
55 | |
56 | Entities | |
57 | ^^^^^^^^ | |
58 | ||
9303c9d5 | 59 | Entities are represented by a struct media_entity |
74604b73 MCC |
60 | instance, defined in ``include/media/media-entity.h``. The structure is usually |
61 | embedded into a higher-level structure, such as | |
041d8211 | 62 | :c:type:`v4l2_subdev` or :c:type:`video_device` |
74604b73 | 63 | instances, although drivers can allocate entities directly. |
684ffa2d MCC |
64 | |
65 | Drivers initialize entity pads by calling | |
7b998bae | 66 | :c:func:`media_entity_pads_init()`. |
684ffa2d MCC |
67 | |
68 | Drivers register entities with a media device by calling | |
7b998bae | 69 | :c:func:`media_device_register_entity()` |
adf48e3f | 70 | and unregistered by calling |
7b998bae | 71 | :c:func:`media_device_unregister_entity()`. |
684ffa2d MCC |
72 | |
73 | Interfaces | |
74 | ^^^^^^^^^^ | |
75 | ||
74604b73 | 76 | Interfaces are represented by a |
9303c9d5 | 77 | struct media_interface instance, defined in |
74604b73 MCC |
78 | ``include/media/media-entity.h``. Currently, only one type of interface is |
79 | defined: a device node. Such interfaces are represented by a | |
9303c9d5 | 80 | struct media_intf_devnode. |
684ffa2d MCC |
81 | |
82 | Drivers initialize and create device node interfaces by calling | |
7b998bae | 83 | :c:func:`media_devnode_create()` |
684ffa2d | 84 | and remove them by calling: |
7b998bae | 85 | :c:func:`media_devnode_remove()`. |
684ffa2d MCC |
86 | |
87 | Pads | |
88 | ^^^^ | |
9303c9d5 | 89 | Pads are represented by a struct media_pad instance, |
74604b73 MCC |
90 | defined in ``include/media/media-entity.h``. Each entity stores its pads in |
91 | a pads array managed by the entity driver. Drivers usually embed the array in | |
92 | a driver-specific structure. | |
684ffa2d MCC |
93 | |
94 | Pads are identified by their entity and their 0-based index in the pads | |
95 | array. | |
74604b73 | 96 | |
9303c9d5 MCC |
97 | Both information are stored in the struct media_pad, |
98 | making the struct media_pad pointer the canonical way | |
3c7d91ef | 99 | to store and pass link references. |
684ffa2d MCC |
100 | |
101 | Pads have flags that describe the pad capabilities and state. | |
102 | ||
74604b73 MCC |
103 | ``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. |
104 | ``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. | |
105 | ||
106 | .. note:: | |
684ffa2d | 107 | |
74604b73 MCC |
108 | One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must |
109 | be set for each pad. | |
684ffa2d MCC |
110 | |
111 | Links | |
112 | ^^^^^ | |
113 | ||
9303c9d5 | 114 | Links are represented by a struct media_link instance, |
74604b73 | 115 | defined in ``include/media/media-entity.h``. There are two types of links: |
684ffa2d | 116 | |
74604b73 | 117 | **1. pad to pad links**: |
684ffa2d MCC |
118 | |
119 | Associate two entities via their PADs. Each entity has a list that points | |
120 | to all links originating at or targeting any of its pads. | |
121 | A given link is thus stored twice, once in the source entity and once in | |
122 | the target entity. | |
123 | ||
124 | Drivers create pad to pad links by calling: | |
7b998bae MCC |
125 | :c:func:`media_create_pad_link()` and remove with |
126 | :c:func:`media_entity_remove_links()`. | |
684ffa2d | 127 | |
74604b73 | 128 | **2. interface to entity links**: |
684ffa2d MCC |
129 | |
130 | Associate one interface to a Link. | |
131 | ||
132 | Drivers create interface to entity links by calling: | |
7b998bae MCC |
133 | :c:func:`media_create_intf_link()` and remove with |
134 | :c:func:`media_remove_intf_links()`. | |
684ffa2d MCC |
135 | |
136 | .. note:: | |
137 | ||
138 | Links can only be created after having both ends already created. | |
139 | ||
140 | Links have flags that describe the link capabilities and state. The | |
7b998bae MCC |
141 | valid values are described at :c:func:`media_create_pad_link()` and |
142 | :c:func:`media_create_intf_link()`. | |
684ffa2d MCC |
143 | |
144 | Graph traversal | |
145 | ^^^^^^^^^^^^^^^ | |
146 | ||
147 | The media framework provides APIs to iterate over entities in a graph. | |
148 | ||
149 | To iterate over all entities belonging to a media device, drivers can use | |
150 | the media_device_for_each_entity macro, defined in | |
74604b73 MCC |
151 | ``include/media/media-device.h``. |
152 | ||
153 | .. code-block:: c | |
684ffa2d | 154 | |
74604b73 | 155 | struct media_entity *entity; |
684ffa2d | 156 | |
74604b73 MCC |
157 | media_device_for_each_entity(entity, mdev) { |
158 | // entity will point to each entity in turn | |
159 | ... | |
160 | } | |
684ffa2d MCC |
161 | |
162 | Drivers might also need to iterate over all entities in a graph that can be | |
163 | reached only through enabled links starting at a given entity. The media | |
164 | framework provides a depth-first graph traversal API for that purpose. | |
165 | ||
74604b73 MCC |
166 | .. note:: |
167 | ||
168 | Graphs with cycles (whether directed or undirected) are **NOT** | |
169 | supported by the graph traversal API. To prevent infinite loops, the graph | |
170 | traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``, | |
171 | currently defined as 16. | |
684ffa2d MCC |
172 | |
173 | Drivers initiate a graph traversal by calling | |
20b85227 | 174 | :c:func:`media_graph_walk_start()` |
684ffa2d MCC |
175 | |
176 | The graph structure, provided by the caller, is initialized to start graph | |
177 | traversal at the given entity. | |
178 | ||
179 | Drivers can then retrieve the next entity by calling | |
20b85227 | 180 | :c:func:`media_graph_walk_next()` |
684ffa2d | 181 | |
74604b73 | 182 | When the graph traversal is complete the function will return ``NULL``. |
684ffa2d MCC |
183 | |
184 | Graph traversal can be interrupted at any moment. No cleanup function call | |
185 | is required and the graph structure can be freed normally. | |
186 | ||
187 | Helper functions can be used to find a link between two given pads, or a pad | |
188 | connected to another pad through an enabled link | |
bb85604b | 189 | (:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`, |
03b28286 LP |
190 | :c:func:`media_entity_remote_source_pad_unique()` and |
191 | :c:func:`media_pad_remote_pad_unique()`). | |
684ffa2d MCC |
192 | |
193 | Use count and power handling | |
194 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
195 | ||
196 | Due to the wide differences between drivers regarding power management | |
197 | needs, the media controller does not implement power management. However, | |
9303c9d5 | 198 | the struct media_entity includes a ``use_count`` |
74604b73 | 199 | field that media drivers |
684ffa2d MCC |
200 | can use to track the number of users of every entity for power management |
201 | needs. | |
202 | ||
74604b73 MCC |
203 | The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by |
204 | media drivers and must not be | |
684ffa2d | 205 | touched by entity drivers. Access to the field must be protected by the |
74604b73 | 206 | :c:type:`media_device`.\ ``graph_mutex`` lock. |
684ffa2d MCC |
207 | |
208 | Links setup | |
209 | ^^^^^^^^^^^ | |
210 | ||
211 | Link properties can be modified at runtime by calling | |
7b998bae | 212 | :c:func:`media_entity_setup_link()`. |
684ffa2d MCC |
213 | |
214 | Pipelines and media streams | |
215 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
216 | ||
b558ce56 TV |
217 | A media stream is a stream of pixels or metadata originating from one or more |
218 | source devices (such as a sensors) and flowing through media entity pads | |
219 | towards the final sinks. The stream can be modified on the route by the | |
220 | devices (e.g. scaling or pixel format conversions), or it can be split into | |
221 | multiple branches, or multiple branches can be merged. | |
222 | ||
223 | A media pipeline is a set of media streams which are interdependent. This | |
224 | interdependency can be caused by the hardware (e.g. configuration of a second | |
225 | stream cannot be changed if the first stream has been enabled) or by the driver | |
226 | due to the software design. Most commonly a media pipeline consists of a single | |
227 | stream which does not branch. | |
228 | ||
684ffa2d MCC |
229 | When starting streaming, drivers must notify all entities in the pipeline to |
230 | prevent link states from being modified during streaming by calling | |
20b85227 | 231 | :c:func:`media_pipeline_start()`. |
684ffa2d | 232 | |
ae219872 | 233 | The function will mark all the pads which are part of the pipeline as streaming. |
684ffa2d | 234 | |
b5163545 LP |
235 | The struct media_pipeline instance pointed to by the pipe argument will be |
236 | stored in every pad in the pipeline. Drivers should embed the struct | |
237 | media_pipeline in higher-level pipeline structures and can then access the | |
238 | pipeline through the struct media_pad pipe field. | |
684ffa2d | 239 | |
20b85227 | 240 | Calls to :c:func:`media_pipeline_start()` can be nested. |
74604b73 | 241 | The pipeline pointer must be identical for all nested calls to the function. |
684ffa2d | 242 | |
20b85227 | 243 | :c:func:`media_pipeline_start()` may return an error. In that case, |
74604b73 | 244 | it will clean up any of the changes it did by itself. |
684ffa2d MCC |
245 | |
246 | When stopping the stream, drivers must notify the entities with | |
20b85227 | 247 | :c:func:`media_pipeline_stop()`. |
684ffa2d | 248 | |
20b85227 SA |
249 | If multiple calls to :c:func:`media_pipeline_start()` have been |
250 | made the same number of :c:func:`media_pipeline_stop()` calls | |
74604b73 MCC |
251 | are required to stop streaming. |
252 | The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last | |
253 | nested stop call. | |
684ffa2d | 254 | |
74604b73 | 255 | Link configuration will fail with ``-EBUSY`` by default if either end of the |
684ffa2d | 256 | link is a streaming entity. Links that can be modified while streaming must |
74604b73 | 257 | be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. |
684ffa2d MCC |
258 | |
259 | If other operations need to be disallowed on streaming entities (such as | |
260 | changing entities configuration parameters) drivers can explicitly check the | |
261 | media_entity stream_count field to find out if an entity is streaming. This | |
262 | operation must be done with the media_device graph_mutex held. | |
263 | ||
264 | Link validation | |
265 | ^^^^^^^^^^^^^^^ | |
266 | ||
20b85227 | 267 | Link validation is performed by :c:func:`media_pipeline_start()` |
74604b73 MCC |
268 | for any entity which has sink pads in the pipeline. The |
269 | :c:type:`media_entity`.\ ``link_validate()`` callback is used for that | |
270 | purpose. In ``link_validate()`` callback, entity driver should check | |
271 | that the properties of the source pad of the connected entity and its own | |
272 | sink pad match. It is up to the type of the entity (and in the end, the | |
273 | properties of the hardware) what matching actually means. | |
684ffa2d MCC |
274 | |
275 | Subsystems should facilitate link validation by providing subsystem specific | |
276 | helper functions to provide easy access for commonly needed information, and | |
277 | in the end provide a way to use driver-specific callbacks. | |
278 | ||
6e1d824e SK |
279 | Media Controller Device Allocator API |
280 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | |
281 | ||
282 | When the media device belongs to more than one driver, the shared media | |
283 | device is allocated with the shared struct device as the key for look ups. | |
284 | ||
285 | The shared media device should stay in registered state until the last | |
286 | driver unregisters it. In addition, the media device should be released when | |
287 | all the references are released. Each driver gets a reference to the media | |
288 | device during probe, when it allocates the media device. If media device is | |
289 | already allocated, the allocate API bumps up the refcount and returns the | |
290 | existing media device. The driver puts the reference back in its disconnect | |
291 | routine when it calls :c:func:`media_device_delete()`. | |
292 | ||
293 | The media device is unregistered and cleaned up from the kref put handler to | |
294 | ensure that the media device stays in registered state until the last driver | |
295 | unregisters the media device. | |
296 | ||
297 | **Driver Usage** | |
298 | ||
299 | Drivers should use the appropriate media-core routines to manage the shared | |
300 | media device life-time handling the two states: | |
301 | 1. allocate -> register -> delete | |
302 | 2. get reference to already registered device -> delete | |
303 | ||
304 | call :c:func:`media_device_delete()` routine to make sure the shared media | |
305 | device delete is handled correctly. | |
306 | ||
307 | **driver probe:** | |
308 | Call :c:func:`media_device_usb_allocate()` to allocate or get a reference | |
309 | Call :c:func:`media_device_register()`, if media devnode isn't registered | |
310 | ||
311 | **driver disconnect:** | |
312 | Call :c:func:`media_device_delete()` to free the media_device. Freeing is | |
313 | handled by the kref put handler. | |
314 | ||
315 | API Definitions | |
316 | ^^^^^^^^^^^^^^^ | |
317 | ||
684ffa2d MCC |
318 | .. kernel-doc:: include/media/media-device.h |
319 | ||
320 | .. kernel-doc:: include/media/media-devnode.h | |
321 | ||
322 | .. kernel-doc:: include/media/media-entity.h | |
496f6f4d SA |
323 | |
324 | .. kernel-doc:: include/media/media-request.h | |
6e1d824e SK |
325 | |
326 | .. kernel-doc:: include/media/media-dev-allocator.h |