Commit | Line | Data |
---|---|---|
059b1c5b | 1 | .. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later |
82559ac0 | 2 | |
5377d91f MH |
3 | .. _subdev: |
4 | ||
5 | ******************** | |
6 | Sub-device Interface | |
7 | ******************** | |
8 | ||
9 | The complex nature of V4L2 devices, where hardware is often made of | |
10 | several integrated circuits that need to interact with each other in a | |
11 | controlled way, leads to complex V4L2 drivers. The drivers usually | |
12 | reflect the hardware model in software, and model the different hardware | |
13 | components as software blocks called sub-devices. | |
14 | ||
15 | V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver | |
16 | implements the media device API, they will automatically inherit from | |
17 | media entities. Applications will be able to enumerate the sub-devices | |
18 | and discover the hardware topology using the media entities, pads and | |
19 | links enumeration API. | |
20 | ||
21 | In addition to make sub-devices discoverable, drivers can also choose to | |
22 | make them directly configurable by applications. When both the | |
23 | sub-device driver and the V4L2 device driver support this, sub-devices | |
24 | will feature a character device node on which ioctls can be called to | |
25 | ||
26 | - query, read and write sub-devices controls | |
27 | ||
28 | - subscribe and unsubscribe to events and retrieve them | |
29 | ||
30 | - negotiate image formats on individual pads | |
31 | ||
32 | Sub-device character device nodes, conventionally named | |
33 | ``/dev/v4l-subdev*``, use major number 81. | |
34 | ||
3fb0ee8b JM |
35 | Drivers may opt to limit the sub-device character devices to only expose |
36 | operations that do not modify the device state. In such a case the sub-devices | |
37 | are referred to as ``read-only`` in the rest of this documentation, and the | |
38 | related restrictions are documented in individual ioctls. | |
39 | ||
5377d91f MH |
40 | |
41 | Controls | |
42 | ======== | |
43 | ||
44 | Most V4L2 controls are implemented by sub-device hardware. Drivers | |
45 | usually merge all controls and expose them through video device nodes. | |
46 | Applications can control all sub-devices through a single interface. | |
47 | ||
48 | Complex devices sometimes implement the same control in different pieces | |
49 | of hardware. This situation is common in embedded platforms, where both | |
50 | sensors and image processing hardware implement identical functions, | |
51 | such as contrast adjustment, white balance or faulty pixels correction. | |
52 | As the V4L2 controls API doesn't support several identical controls in a | |
53 | single device, all but one of the identical controls are hidden. | |
54 | ||
55 | Applications can access those hidden controls through the sub-device | |
56 | node with the V4L2 control API described in :ref:`control`. The ioctls | |
57 | behave identically as when issued on V4L2 device nodes, with the | |
58 | exception that they deal only with controls implemented in the | |
59 | sub-device. | |
60 | ||
61 | Depending on the driver, those controls might also be exposed through | |
62 | one (or several) V4L2 device nodes. | |
63 | ||
64 | ||
65 | Events | |
66 | ====== | |
67 | ||
68 | V4L2 sub-devices can notify applications of events as described in | |
69 | :ref:`event`. The API behaves identically as when used on V4L2 device | |
70 | nodes, with the exception that it only deals with events generated by | |
71 | the sub-device. Depending on the driver, those events might also be | |
72 | reported on one (or several) V4L2 device nodes. | |
73 | ||
74 | ||
75 | .. _pad-level-formats: | |
76 | ||
77 | Pad-level Formats | |
78 | ================= | |
79 | ||
706f8a99 | 80 | .. warning:: |
5377d91f | 81 | |
706f8a99 | 82 | Pad-level formats are only applicable to very complex devices that |
5377d91f MH |
83 | need to expose low-level format configuration to user space. Generic |
84 | V4L2 applications do *not* need to use the API described in this | |
85 | section. | |
86 | ||
706f8a99 | 87 | .. note:: |
5377d91f MH |
88 | |
89 | For the purpose of this section, the term *format* means the | |
90 | combination of media bus data format, frame width and frame height. | |
91 | ||
92 | Image formats are typically negotiated on video capture and output | |
93 | devices using the format and | |
af4a4d0d | 94 | :ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver is |
5377d91f MH |
95 | responsible for configuring every block in the video pipeline according |
96 | to the requested format at the pipeline input and/or output. | |
97 | ||
98 | For complex devices, such as often found in embedded systems, identical | |
99 | image sizes at the output of a pipeline can be achieved using different | |
100 | hardware configurations. One such example is shown on | |
101 | :ref:`pipeline-scaling`, where image scaling can be performed on both | |
102 | the video sensor and the host image processing hardware. | |
103 | ||
104 | ||
105 | .. _pipeline-scaling: | |
106 | ||
8fa1bb50 MCC |
107 | .. kernel-figure:: pipeline.dot |
108 | :alt: pipeline.dot | |
109 | :align: center | |
5377d91f MH |
110 | |
111 | Image Format Negotiation on Pipelines | |
112 | ||
113 | High quality and high speed pipeline configuration | |
114 | ||
115 | ||
116 | ||
117 | The sensor scaler is usually of less quality than the host scaler, but | |
118 | scaling on the sensor is required to achieve higher frame rates. | |
119 | Depending on the use case (quality vs. speed), the pipeline must be | |
120 | configured differently. Applications need to configure the formats at | |
121 | every point in the pipeline explicitly. | |
122 | ||
123 | Drivers that implement the :ref:`media API <media-controller-intro>` | |
124 | can expose pad-level image format configuration to applications. When | |
125 | they do, applications can use the | |
c6f7b0f2 | 126 | :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and |
af4a4d0d | 127 | :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. to |
5377d91f MH |
128 | negotiate formats on a per-pad basis. |
129 | ||
130 | Applications are responsible for configuring coherent parameters on the | |
131 | whole pipeline and making sure that connected pads have compatible | |
132 | formats. The pipeline is checked for formats mismatch at | |
c6f7b0f2 | 133 | :ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` error |
5377d91f MH |
134 | code is then returned if the configuration is invalid. |
135 | ||
136 | Pad-level image format configuration support can be tested by calling | |
7347081e | 137 | the :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad |
cdb4af0f | 138 | 0. If the driver returns an ``EINVAL`` error code pad-level format |
5377d91f MH |
139 | configuration is not supported by the sub-device. |
140 | ||
141 | ||
142 | Format Negotiation | |
143 | ------------------ | |
144 | ||
145 | Acceptable formats on pads can (and usually do) depend on a number of | |
146 | external parameters, such as formats on other pads, active links, or | |
147 | even controls. Finding a combination of formats on all pads in a video | |
148 | pipeline, acceptable to both application and driver, can't rely on | |
149 | formats enumeration only. A format negotiation mechanism is required. | |
150 | ||
151 | Central to the format negotiation mechanism are the get/set format | |
152 | operations. When called with the ``which`` argument set to | |
c6f7b0f2 MCC |
153 | :ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the |
154 | :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and | |
af4a4d0d | 155 | :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate on |
5377d91f MH |
156 | a set of formats parameters that are not connected to the hardware |
157 | configuration. Modifying those 'try' formats leaves the device state | |
158 | untouched (this applies to both the software state stored in the driver | |
159 | and the hardware state stored in the device itself). | |
160 | ||
161 | While not kept as part of the device state, try formats are stored in | |
162 | the sub-device file handles. A | |
c6f7b0f2 | 163 | :ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will return |
5377d91f MH |
164 | the last try format set *on the same sub-device file handle*. Several |
165 | applications querying the same sub-device at the same time will thus not | |
166 | interact with each other. | |
167 | ||
168 | To find out whether a particular format is supported by the device, | |
169 | applications use the | |
af4a4d0d | 170 | :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Drivers |
5377d91f MH |
171 | verify and, if needed, change the requested ``format`` based on device |
172 | requirements and return the possibly modified value. Applications can | |
173 | then choose to try a different format or accept the returned value and | |
174 | continue. | |
175 | ||
176 | Formats returned by the driver during a negotiation iteration are | |
177 | guaranteed to be supported by the device. In particular, drivers | |
178 | guarantee that a returned format will not be further changed if passed | |
af4a4d0d | 179 | to an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is |
5377d91f MH |
180 | (as long as external parameters, such as formats on other pads or links' |
181 | configuration are not changed). | |
182 | ||
183 | Drivers automatically propagate formats inside sub-devices. When a try | |
184 | or active format is set on a pad, corresponding formats on other pads of | |
185 | the same sub-device can be modified by the driver. Drivers are free to | |
186 | modify formats as required by the device. However, they should comply | |
187 | with the following rules when possible: | |
188 | ||
189 | - Formats should be propagated from sink pads to source pads. Modifying | |
190 | a format on a source pad should not modify the format on any sink | |
191 | pad. | |
192 | ||
193 | - Sub-devices that scale frames using variable scaling factors should | |
194 | reset the scale factors to default values when sink pads formats are | |
195 | modified. If the 1:1 scaling ratio is supported, this means that | |
196 | source pads formats should be reset to the sink pads formats. | |
197 | ||
198 | Formats are not propagated across links, as that would involve | |
199 | propagating them from one sub-device file handle to another. | |
200 | Applications must then take care to configure both ends of every link | |
201 | explicitly with compatible formats. Identical formats on the two ends of | |
202 | a link are guaranteed to be compatible. Drivers are free to accept | |
203 | different formats matching device requirements as being compatible. | |
204 | ||
205 | :ref:`sample-pipeline-config` shows a sample configuration sequence | |
206 | for the pipeline described in :ref:`pipeline-scaling` (table columns | |
207 | list entity names and pad numbers). | |
208 | ||
209 | ||
3772b56a MCC |
210 | .. raw:: latex |
211 | ||
94fa831e | 212 | \scriptsize |
3772b56a | 213 | |
70b074df | 214 | .. tabularcolumns:: |p{2.0cm}|p{2.3cm}|p{2.3cm}|p{2.3cm}|p{2.3cm}|p{2.3cm}|p{2.3cm}| |
3772b56a | 215 | |
fa92b04d MCC |
216 | .. _sample-pipeline-config: |
217 | ||
5377d91f MH |
218 | .. flat-table:: Sample Pipeline Configuration |
219 | :header-rows: 1 | |
220 | :stub-columns: 0 | |
3772b56a | 221 | :widths: 5 5 5 5 5 5 5 |
5377d91f | 222 | |
c2b66caf | 223 | * - |
94fa831e MCC |
224 | - Sensor/0 |
225 | ||
226 | format | |
227 | - Frontend/0 | |
228 | ||
229 | format | |
230 | - Frontend/1 | |
231 | ||
232 | format | |
233 | - Scaler/0 | |
234 | ||
235 | format | |
236 | - Scaler/0 | |
237 | ||
238 | compose selection rectangle | |
239 | - Scaler/1 | |
240 | ||
241 | format | |
c2b66caf | 242 | * - Initial state |
94fa831e MCC |
243 | - 2048x1536 |
244 | ||
245 | SGRBG8_1X8 | |
c2b66caf LP |
246 | - (default) |
247 | - (default) | |
248 | - (default) | |
249 | - (default) | |
250 | - (default) | |
251 | * - Configure frontend sink format | |
94fa831e MCC |
252 | - 2048x1536 |
253 | ||
254 | SGRBG8_1X8 | |
255 | - *2048x1536* | |
256 | ||
257 | *SGRBG8_1X8* | |
258 | - *2046x1534* | |
259 | ||
260 | *SGRBG8_1X8* | |
c2b66caf LP |
261 | - (default) |
262 | - (default) | |
263 | - (default) | |
264 | * - Configure scaler sink format | |
94fa831e MCC |
265 | - 2048x1536 |
266 | ||
267 | SGRBG8_1X8 | |
268 | - 2048x1536 | |
269 | ||
270 | SGRBG8_1X8 | |
271 | - 2046x1534 | |
272 | ||
273 | SGRBG8_1X8 | |
274 | - *2046x1534* | |
275 | ||
276 | *SGRBG8_1X8* | |
c2b66caf | 277 | - *0,0/2046x1534* |
94fa831e MCC |
278 | - *2046x1534* |
279 | ||
280 | *SGRBG8_1X8* | |
c2b66caf | 281 | * - Configure scaler sink compose selection |
94fa831e MCC |
282 | - 2048x1536 |
283 | ||
284 | SGRBG8_1X8 | |
285 | - 2048x1536 | |
286 | ||
287 | SGRBG8_1X8 | |
288 | - 2046x1534 | |
289 | ||
290 | SGRBG8_1X8 | |
291 | - 2046x1534 | |
292 | ||
293 | SGRBG8_1X8 | |
c2b66caf | 294 | - *0,0/1280x960* |
94fa831e MCC |
295 | - *1280x960* |
296 | ||
297 | *SGRBG8_1X8* | |
5377d91f | 298 | |
3772b56a | 299 | .. raw:: latex |
5377d91f | 300 | |
70b074df | 301 | \normalsize |
5377d91f MH |
302 | |
303 | 1. Initial state. The sensor source pad format is set to its native 3MP | |
304 | size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the | |
305 | host frontend and scaler sink and source pads have the default | |
306 | values, as well as the compose rectangle on the scaler's sink pad. | |
307 | ||
308 | 2. The application configures the frontend sink pad format's size to | |
309 | 2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The | |
310 | driver propagates the format to the frontend source pad. | |
311 | ||
312 | 3. The application configures the scaler sink pad format's size to | |
313 | 2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to | |
314 | match the frontend source size and media bus code. The media bus code | |
315 | on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver | |
316 | propagates the size to the compose selection rectangle on the | |
317 | scaler's sink pad, and the format to the scaler source pad. | |
318 | ||
319 | 4. The application configures the size of the compose selection | |
320 | rectangle of the scaler's sink pad 1280x960. The driver propagates | |
321 | the size to the scaler's source pad format. | |
322 | ||
323 | When satisfied with the try results, applications can set the active | |
324 | formats by setting the ``which`` argument to | |
325 | ``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try | |
326 | formats by drivers. To avoid modifying the hardware state during format | |
327 | negotiation, applications should negotiate try formats first and then | |
328 | modify the active settings using the try formats returned during the | |
329 | last negotiation iteration. This guarantees that the active format will | |
330 | be applied as-is by the driver without being modified. | |
331 | ||
332 | ||
333 | .. _v4l2-subdev-selections: | |
334 | ||
335 | Selections: cropping, scaling and composition | |
336 | --------------------------------------------- | |
337 | ||
338 | Many sub-devices support cropping frames on their input or output pads | |
339 | (or possible even on both). Cropping is used to select the area of | |
340 | interest in an image, typically on an image sensor or a video decoder. | |
341 | It can also be used as part of digital zoom implementations to select | |
342 | the area of the image that will be scaled up. | |
343 | ||
344 | Crop settings are defined by a crop rectangle and represented in a | |
e8be7e97 | 345 | struct :c:type:`v4l2_rect` by the coordinates of the top |
5377d91f MH |
346 | left corner and the rectangle size. Both the coordinates and sizes are |
347 | expressed in pixels. | |
348 | ||
349 | As for pad formats, drivers store try and active rectangles for the | |
350 | selection targets :ref:`v4l2-selections-common`. | |
351 | ||
352 | On sink pads, cropping is applied relative to the current pad format. | |
353 | The pad format represents the image size as received by the sub-device | |
354 | from the previous block in the pipeline, and the crop rectangle | |
355 | represents the sub-image that will be transmitted further inside the | |
356 | sub-device for processing. | |
357 | ||
358 | The scaling operation changes the size of the image by scaling it to new | |
359 | dimensions. The scaling ratio isn't specified explicitly, but is implied | |
360 | from the original and scaled image sizes. Both sizes are represented by | |
e8be7e97 | 361 | struct :c:type:`v4l2_rect`. |
5377d91f MH |
362 | |
363 | Scaling support is optional. When supported by a subdev, the crop | |
364 | rectangle on the subdev's sink pad is scaled to the size configured | |
365 | using the | |
af4a4d0d | 366 | :ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTL |
5377d91f MH |
367 | using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the |
368 | subdev supports scaling but not composing, the top and left values are | |
369 | not used and must always be set to zero. | |
370 | ||
371 | On source pads, cropping is similar to sink pads, with the exception | |
372 | that the source size from which the cropping is performed, is the | |
373 | COMPOSE rectangle on the sink pad. In both sink and source pads, the | |
374 | crop rectangle must be entirely contained inside the source image size | |
375 | for the crop operation. | |
376 | ||
377 | The drivers should always use the closest possible rectangle the user | |
378 | requests on all selection targets, unless specifically told otherwise. | |
379 | ``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round | |
380 | the image size either up or down. :ref:`v4l2-selection-flags` | |
381 | ||
382 | ||
383 | Types of selection targets | |
384 | -------------------------- | |
385 | ||
386 | ||
387 | Actual targets | |
388 | ^^^^^^^^^^^^^^ | |
389 | ||
390 | Actual targets (without a postfix) reflect the actual hardware | |
391 | configuration at any point of time. There is a BOUNDS target | |
392 | corresponding to every actual target. | |
393 | ||
394 | ||
395 | BOUNDS targets | |
396 | ^^^^^^^^^^^^^^ | |
397 | ||
398 | BOUNDS targets is the smallest rectangle that contains all valid actual | |
399 | rectangles. It may not be possible to set the actual rectangle as large | |
400 | as the BOUNDS rectangle, however. This may be because e.g. a sensor's | |
401 | pixel array is not rectangular but cross-shaped or round. The maximum | |
402 | size may also be smaller than the BOUNDS rectangle. | |
403 | ||
404 | ||
405 | Order of configuration and format propagation | |
406 | --------------------------------------------- | |
407 | ||
408 | Inside subdevs, the order of image processing steps will always be from | |
409 | the sink pad towards the source pad. This is also reflected in the order | |
410 | in which the configuration must be performed by the user: the changes | |
411 | made will be propagated to any subsequent stages. If this behaviour is | |
412 | not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This | |
413 | flag causes no propagation of the changes are allowed in any | |
414 | circumstances. This may also cause the accessed rectangle to be adjusted | |
415 | by the driver, depending on the properties of the underlying hardware. | |
416 | ||
417 | The coordinates to a step always refer to the actual size of the | |
74dcb29a | 418 | previous step. The exception to this rule is the sink compose |
5377d91f MH |
419 | rectangle, which refers to the sink compose bounds rectangle --- if it |
420 | is supported by the hardware. | |
421 | ||
422 | 1. Sink pad format. The user configures the sink pad format. This format | |
423 | defines the parameters of the image the entity receives through the | |
424 | pad for further processing. | |
425 | ||
426 | 2. Sink pad actual crop selection. The sink pad crop defines the crop | |
427 | performed to the sink pad format. | |
428 | ||
429 | 3. Sink pad actual compose selection. The size of the sink pad compose | |
430 | rectangle defines the scaling ratio compared to the size of the sink | |
431 | pad crop rectangle. The location of the compose rectangle specifies | |
432 | the location of the actual sink compose rectangle in the sink compose | |
433 | bounds rectangle. | |
434 | ||
435 | 4. Source pad actual crop selection. Crop on the source pad defines crop | |
436 | performed to the image in the sink compose bounds rectangle. | |
437 | ||
438 | 5. Source pad format. The source pad format defines the output pixel | |
439 | format of the subdev, as well as the other parameters with the | |
440 | exception of the image width and height. Width and height are defined | |
441 | by the size of the source pad actual crop selection. | |
442 | ||
443 | Accessing any of the above rectangles not supported by the subdev will | |
444 | return ``EINVAL``. Any rectangle referring to a previous unsupported | |
445 | rectangle coordinates will instead refer to the previous supported | |
446 | rectangle. For example, if sink crop is not supported, the compose | |
447 | selection will refer to the sink pad format dimensions instead. | |
448 | ||
449 | ||
450 | .. _subdev-image-processing-crop: | |
451 | ||
8fa1bb50 MCC |
452 | .. kernel-figure:: subdev-image-processing-crop.svg |
453 | :alt: subdev-image-processing-crop.svg | |
454 | :align: center | |
5377d91f | 455 | |
c6f7b0f2 | 456 | **Figure 4.5. Image processing in subdevs: simple crop example** |
5377d91f MH |
457 | |
458 | In the above example, the subdev supports cropping on its sink pad. To | |
459 | configure it, the user sets the media bus format on the subdev's sink | |
460 | pad. Now the actual crop rectangle can be set on the sink pad --- the | |
461 | location and size of this rectangle reflect the location and size of a | |
462 | rectangle to be cropped from the sink format. The size of the sink crop | |
463 | rectangle will also be the size of the format of the subdev's source | |
464 | pad. | |
465 | ||
466 | ||
467 | .. _subdev-image-processing-scaling-multi-source: | |
468 | ||
8fa1bb50 MCC |
469 | .. kernel-figure:: subdev-image-processing-scaling-multi-source.svg |
470 | :alt: subdev-image-processing-scaling-multi-source.svg | |
471 | :align: center | |
5377d91f | 472 | |
c6f7b0f2 | 473 | **Figure 4.6. Image processing in subdevs: scaling with multiple sources** |
5377d91f MH |
474 | |
475 | In this example, the subdev is capable of first cropping, then scaling | |
476 | and finally cropping for two source pads individually from the resulting | |
477 | scaled image. The location of the scaled image in the cropped image is | |
478 | ignored in sink compose target. Both of the locations of the source crop | |
479 | rectangles refer to the sink scaling rectangle, independently cropping | |
480 | an area at location specified by the source crop rectangle from it. | |
481 | ||
482 | ||
483 | .. _subdev-image-processing-full: | |
484 | ||
8fa1bb50 MCC |
485 | .. kernel-figure:: subdev-image-processing-full.svg |
486 | :alt: subdev-image-processing-full.svg | |
5377d91f MH |
487 | :align: center |
488 | ||
c6f7b0f2 | 489 | **Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources** |
5377d91f MH |
490 | |
491 | The subdev driver supports two sink pads and two source pads. The images | |
492 | from both of the sink pads are individually cropped, then scaled and | |
493 | further composed on the composition bounds rectangle. From that, two | |
494 | independent streams are cropped and sent out of the subdev from the | |
495 | source pads. | |
496 | ||
497 | ||
498 | .. toctree:: | |
499 | :maxdepth: 1 | |
500 | ||
501 | subdev-formats |