Commit | Line | Data |
---|---|---|
176fb0d1 LP |
1 | Linux kernel media framework |
2 | ============================ | |
3 | ||
4 | This document describes the Linux kernel media framework, its data structures, | |
5 | functions and their usage. | |
6 | ||
7 | ||
8 | Introduction | |
9 | ------------ | |
10 | ||
11 | The media controller API is documented in DocBook format in | |
395cf969 PB |
12 | Documentation/DocBook/media/v4l/media-controller.xml. This document will focus |
13 | on the kernel-side implementation of the media framework. | |
176fb0d1 LP |
14 | |
15 | ||
53e269c1 LP |
16 | Abstract media device model |
17 | --------------------------- | |
18 | ||
19 | Discovering a device internal topology, and configuring it at runtime, is one | |
20 | of the goals of the media framework. To achieve this, hardware devices are | |
f884ab15 | 21 | modelled as an oriented graph of building blocks called entities connected |
53e269c1 LP |
22 | through pads. |
23 | ||
24 | An entity is a basic media hardware building block. It can correspond to | |
25 | a large variety of logical blocks such as physical hardware devices | |
26 | (CMOS sensor for instance), logical hardware devices (a building block | |
27 | in a System-on-Chip image processing pipeline), DMA channels or physical | |
28 | connectors. | |
29 | ||
30 | A pad is a connection endpoint through which an entity can interact with | |
31 | other entities. Data (not restricted to video) produced by an entity | |
32 | flows from the entity's output to one or more entity inputs. Pads should | |
33 | not be confused with physical pins at chip boundaries. | |
34 | ||
35 | A link is a point-to-point oriented connection between two pads, either | |
36 | on the same entity or on different entities. Data flows from a source | |
37 | pad to a sink pad. | |
38 | ||
39 | ||
176fb0d1 LP |
40 | Media device |
41 | ------------ | |
42 | ||
43 | A media device is represented by a struct media_device instance, defined in | |
44 | include/media/media-device.h. Allocation of the structure is handled by the | |
45 | media device driver, usually by embedding the media_device instance in a | |
46 | larger driver-specific structure. | |
47 | ||
48 | Drivers register media device instances by calling | |
49 | ||
50 | media_device_register(struct media_device *mdev); | |
51 | ||
52 | The caller is responsible for initializing the media_device structure before | |
53 | registration. The following fields must be set: | |
54 | ||
55 | - dev must point to the parent device (usually a pci_dev, usb_interface or | |
56 | platform_device instance). | |
57 | ||
58 | - model must be filled with the device model name as a NUL-terminated UTF-8 | |
59 | string. The device/model revision must not be stored in this field. | |
60 | ||
61 | The following fields are optional: | |
62 | ||
63 | - serial is a unique serial number stored as a NUL-terminated ASCII string. | |
64 | The field is big enough to store a GUID in text form. If the hardware | |
65 | doesn't provide a unique serial number this field must be left empty. | |
66 | ||
67 | - bus_info represents the location of the device in the system as a | |
68 | NUL-terminated ASCII string. For PCI/PCIe devices bus_info must be set to | |
69 | "PCI:" (or "PCIe:") followed by the value of pci_name(). For USB devices, | |
70 | the usb_make_path() function must be used. This field is used by | |
71 | applications to distinguish between otherwise identical devices that don't | |
72 | provide a serial number. | |
73 | ||
74 | - hw_revision is the hardware device revision in a driver-specific format. | |
75 | When possible the revision should be formatted with the KERNEL_VERSION | |
76 | macro. | |
77 | ||
78 | - driver_version is formatted with the KERNEL_VERSION macro. The version | |
79 | minor must be incremented when new features are added to the userspace API | |
80 | without breaking binary compatibility. The version major must be | |
81 | incremented when binary compatibility is broken. | |
82 | ||
83 | Upon successful registration a character device named media[0-9]+ is created. | |
84 | The device major and minor numbers are dynamic. The model name is exported as | |
85 | a sysfs attribute. | |
86 | ||
87 | Drivers unregister media device instances by calling | |
88 | ||
89 | media_device_unregister(struct media_device *mdev); | |
90 | ||
91 | Unregistering a media device that hasn't been registered is *NOT* safe. | |
53e269c1 LP |
92 | |
93 | ||
94 | Entities, pads and links | |
95 | ------------------------ | |
96 | ||
97 | - Entities | |
98 | ||
99 | Entities are represented by a struct media_entity instance, defined in | |
100 | include/media/media-entity.h. The structure is usually embedded into a | |
101 | higher-level structure, such as a v4l2_subdev or video_device instance, | |
102 | although drivers can allocate entities directly. | |
103 | ||
104 | Drivers initialize entities by calling | |
105 | ||
106 | media_entity_init(struct media_entity *entity, u16 num_pads, | |
107 | struct media_pad *pads, u16 extra_links); | |
108 | ||
109 | The media_entity name, type, flags, revision and group_id fields can be | |
110 | initialized before or after calling media_entity_init. Entities embedded in | |
111 | higher-level standard structures can have some of those fields set by the | |
112 | higher-level framework. | |
113 | ||
114 | As the number of pads is known in advance, the pads array is not allocated | |
115 | dynamically but is managed by the entity driver. Most drivers will embed the | |
116 | pads array in a driver-specific structure, avoiding dynamic allocation. | |
117 | ||
118 | Drivers must set the direction of every pad in the pads array before calling | |
119 | media_entity_init. The function will initialize the other pads fields. | |
120 | ||
121 | Unlike the number of pads, the total number of links isn't always known in | |
122 | advance by the entity driver. As an initial estimate, media_entity_init | |
123 | pre-allocates a number of links equal to the number of pads plus an optional | |
124 | number of extra links. The links array will be reallocated if it grows beyond | |
125 | the initial estimate. | |
126 | ||
127 | Drivers register entities with a media device by calling | |
128 | ||
129 | media_device_register_entity(struct media_device *mdev, | |
130 | struct media_entity *entity); | |
131 | ||
132 | Entities are identified by a unique positive integer ID. Drivers can provide an | |
133 | ID by filling the media_entity id field prior to registration, or request the | |
134 | media controller framework to assign an ID automatically. Drivers that provide | |
135 | IDs manually must ensure that all IDs are unique. IDs are not guaranteed to be | |
136 | contiguous even when they are all assigned automatically by the framework. | |
137 | ||
138 | Drivers unregister entities by calling | |
139 | ||
140 | media_device_unregister_entity(struct media_entity *entity); | |
141 | ||
142 | Unregistering an entity will not change the IDs of the other entities, and the | |
143 | ID will never be reused for a newly registered entity. | |
144 | ||
145 | When a media device is unregistered, all its entities are unregistered | |
146 | automatically. No manual entities unregistration is then required. | |
147 | ||
148 | Drivers free resources associated with an entity by calling | |
149 | ||
150 | media_entity_cleanup(struct media_entity *entity); | |
151 | ||
152 | This function must be called during the cleanup phase after unregistering the | |
153 | entity. Note that the media_entity instance itself must be freed explicitly by | |
154 | the driver if required. | |
155 | ||
156 | Entities have flags that describe the entity capabilities and state. | |
157 | ||
158 | MEDIA_ENT_FL_DEFAULT indicates the default entity for a given type. | |
159 | This can be used to report the default audio and video devices or the | |
160 | default camera sensor. | |
161 | ||
162 | Logical entity groups can be defined by setting the group ID of all member | |
163 | entities to the same non-zero value. An entity group serves no purpose in the | |
164 | kernel, but is reported to userspace during entities enumeration. The group_id | |
165 | field belongs to the media device driver and must not by touched by entity | |
166 | drivers. | |
167 | ||
168 | Media device drivers should define groups if several entities are logically | |
169 | bound together. Example usages include reporting | |
170 | ||
171 | - ALSA, VBI and video nodes that carry the same media stream | |
172 | - lens and flash controllers associated with a sensor | |
173 | ||
174 | - Pads | |
175 | ||
176 | Pads are represented by a struct media_pad instance, defined in | |
177 | include/media/media-entity.h. Each entity stores its pads in a pads array | |
178 | managed by the entity driver. Drivers usually embed the array in a | |
179 | driver-specific structure. | |
180 | ||
181 | Pads are identified by their entity and their 0-based index in the pads array. | |
182 | Both information are stored in the media_pad structure, making the media_pad | |
183 | pointer the canonical way to store and pass link references. | |
184 | ||
185 | Pads have flags that describe the pad capabilities and state. | |
186 | ||
187 | MEDIA_PAD_FL_SINK indicates that the pad supports sinking data. | |
188 | MEDIA_PAD_FL_SOURCE indicates that the pad supports sourcing data. | |
189 | ||
190 | One and only one of MEDIA_PAD_FL_SINK and MEDIA_PAD_FL_SOURCE must be set for | |
191 | each pad. | |
192 | ||
193 | - Links | |
194 | ||
195 | Links are represented by a struct media_link instance, defined in | |
196 | include/media/media-entity.h. Each entity stores all links originating at or | |
25985edc | 197 | targeting any of its pads in a links array. A given link is thus stored |
53e269c1 LP |
198 | twice, once in the source entity and once in the target entity. The array is |
199 | pre-allocated and grows dynamically as needed. | |
200 | ||
201 | Drivers create links by calling | |
202 | ||
203 | media_entity_create_link(struct media_entity *source, u16 source_pad, | |
204 | struct media_entity *sink, u16 sink_pad, | |
205 | u32 flags); | |
206 | ||
207 | An entry in the link array of each entity is allocated and stores pointers | |
208 | to source and sink pads. | |
209 | ||
210 | Links have flags that describe the link capabilities and state. | |
211 | ||
212 | MEDIA_LNK_FL_ENABLED indicates that the link is enabled and can be used | |
213 | to transfer media data. When two or more links target a sink pad, only | |
214 | one of them can be enabled at a time. | |
215 | MEDIA_LNK_FL_IMMUTABLE indicates that the link enabled state can't be | |
216 | modified at runtime. If MEDIA_LNK_FL_IMMUTABLE is set, then | |
217 | MEDIA_LNK_FL_ENABLED must also be set since an immutable link is always | |
218 | enabled. | |
a5ccc48a SA |
219 | |
220 | ||
221 | Graph traversal | |
222 | --------------- | |
223 | ||
224 | The media framework provides APIs to iterate over entities in a graph. | |
225 | ||
226 | To iterate over all entities belonging to a media device, drivers can use the | |
227 | media_device_for_each_entity macro, defined in include/media/media-device.h. | |
228 | ||
229 | struct media_entity *entity; | |
230 | ||
231 | media_device_for_each_entity(entity, mdev) { | |
232 | /* entity will point to each entity in turn */ | |
233 | ... | |
234 | } | |
235 | ||
236 | Drivers might also need to iterate over all entities in a graph that can be | |
237 | reached only through enabled links starting at a given entity. The media | |
238 | framework provides a depth-first graph traversal API for that purpose. | |
239 | ||
240 | Note that graphs with cycles (whether directed or undirected) are *NOT* | |
241 | supported by the graph traversal API. To prevent infinite loops, the graph | |
242 | traversal code limits the maximum depth to MEDIA_ENTITY_ENUM_MAX_DEPTH, | |
243 | currently defined as 16. | |
244 | ||
245 | Drivers initiate a graph traversal by calling | |
246 | ||
247 | media_entity_graph_walk_start(struct media_entity_graph *graph, | |
248 | struct media_entity *entity); | |
249 | ||
250 | The graph structure, provided by the caller, is initialized to start graph | |
251 | traversal at the given entity. | |
252 | ||
253 | Drivers can then retrieve the next entity by calling | |
254 | ||
255 | media_entity_graph_walk_next(struct media_entity_graph *graph); | |
256 | ||
257 | When the graph traversal is complete the function will return NULL. | |
258 | ||
259 | Graph traversal can be interrupted at any moment. No cleanup function call is | |
260 | required and the graph structure can be freed normally. | |
503c3d82 | 261 | |
97548ed4 LP |
262 | Helper functions can be used to find a link between two given pads, or a pad |
263 | connected to another pad through an enabled link | |
264 | ||
265 | media_entity_find_link(struct media_pad *source, | |
266 | struct media_pad *sink); | |
267 | ||
1bddf1b3 | 268 | media_entity_remote_pad(struct media_pad *pad); |
97548ed4 LP |
269 | |
270 | Refer to the kerneldoc documentation for more information. | |
271 | ||
503c3d82 LP |
272 | |
273 | Use count and power handling | |
274 | ---------------------------- | |
275 | ||
276 | Due to the wide differences between drivers regarding power management needs, | |
277 | the media controller does not implement power management. However, the | |
278 | media_entity structure includes a use_count field that media drivers can use to | |
279 | track the number of users of every entity for power management needs. | |
280 | ||
281 | The use_count field is owned by media drivers and must not be touched by entity | |
282 | drivers. Access to the field must be protected by the media device graph_mutex | |
283 | lock. | |
97548ed4 LP |
284 | |
285 | ||
286 | Links setup | |
287 | ----------- | |
288 | ||
289 | Link properties can be modified at runtime by calling | |
290 | ||
291 | media_entity_setup_link(struct media_link *link, u32 flags); | |
292 | ||
293 | The flags argument contains the requested new link flags. | |
294 | ||
295 | The only configurable property is the ENABLED link flag to enable/disable a | |
296 | link. Links marked with the IMMUTABLE link flag can not be enabled or disabled. | |
297 | ||
298 | When a link is enabled or disabled, the media framework calls the | |
299 | link_setup operation for the two entities at the source and sink of the link, | |
300 | in that order. If the second link_setup call fails, another link_setup call is | |
301 | made on the first entity to restore the original link flags. | |
302 | ||
303 | Media device drivers can be notified of link setup operations by setting the | |
304 | media_device::link_notify pointer to a callback function. If provided, the | |
305 | notification callback will be called before enabling and after disabling | |
306 | links. | |
307 | ||
308 | Entity drivers must implement the link_setup operation if any of their links | |
309 | is non-immutable. The operation must either configure the hardware or store | |
310 | the configuration information to be applied later. | |
311 | ||
312 | Link configuration must not have any side effect on other links. If an enabled | |
3751e329 | 313 | link at a sink pad prevents another link at the same pad from being enabled, |
97548ed4 LP |
314 | the link_setup operation must return -EBUSY and can't implicitly disable the |
315 | first enabled link. | |
e02188c9 LP |
316 | |
317 | ||
318 | Pipelines and media streams | |
319 | --------------------------- | |
320 | ||
321 | When starting streaming, drivers must notify all entities in the pipeline to | |
322 | prevent link states from being modified during streaming by calling | |
323 | ||
324 | media_entity_pipeline_start(struct media_entity *entity, | |
325 | struct media_pipeline *pipe); | |
326 | ||
327 | The function will mark all entities connected to the given entity through | |
328 | enabled links, either directly or indirectly, as streaming. | |
329 | ||
330 | The media_pipeline instance pointed to by the pipe argument will be stored in | |
331 | every entity in the pipeline. Drivers should embed the media_pipeline structure | |
332 | in higher-level pipeline structures and can then access the pipeline through | |
333 | the media_entity pipe field. | |
334 | ||
335 | Calls to media_entity_pipeline_start() can be nested. The pipeline pointer must | |
336 | be identical for all nested calls to the function. | |
337 | ||
af88be38 | 338 | media_entity_pipeline_start() may return an error. In that case, it will |
c240ac9b | 339 | clean up any of the changes it did by itself. |
af88be38 | 340 | |
e02188c9 LP |
341 | When stopping the stream, drivers must notify the entities with |
342 | ||
343 | media_entity_pipeline_stop(struct media_entity *entity); | |
344 | ||
345 | If multiple calls to media_entity_pipeline_start() have been made the same | |
346 | number of media_entity_pipeline_stop() calls are required to stop streaming. The | |
347 | media_entity pipe field is reset to NULL on the last nested stop call. | |
348 | ||
349 | Link configuration will fail with -EBUSY by default if either end of the link is | |
350 | a streaming entity. Links that can be modified while streaming must be marked | |
351 | with the MEDIA_LNK_FL_DYNAMIC flag. | |
352 | ||
353 | If other operations need to be disallowed on streaming entities (such as | |
25985edc | 354 | changing entities configuration parameters) drivers can explicitly check the |
e02188c9 LP |
355 | media_entity stream_count field to find out if an entity is streaming. This |
356 | operation must be done with the media_device graph_mutex held. | |
af88be38 SA |
357 | |
358 | ||
359 | Link validation | |
360 | --------------- | |
361 | ||
362 | Link validation is performed by media_entity_pipeline_start() for any | |
363 | entity which has sink pads in the pipeline. The | |
364 | media_entity::link_validate() callback is used for that purpose. In | |
365 | link_validate() callback, entity driver should check that the properties of | |
366 | the source pad of the connected entity and its own sink pad match. It is up | |
367 | to the type of the entity (and in the end, the properties of the hardware) | |
368 | what matching actually means. | |
369 | ||
370 | Subsystems should facilitate link validation by providing subsystem specific | |
371 | helper functions to provide easy access for commonly needed information, and | |
372 | in the end provide a way to use driver-specific callbacks. |