Support client usage notification via V4l2 Event API

[You-Sheng Yang]

[SRU Justification]

[Impact]

On a MIPI camera through Intel IPU6 platform that its raw V4L2 loopback interface is preserved for Intel Camera HAL libraries, a relay daemon + v4l2loopback is used to allow the usage of legacy V4L2 based apps.

By design, the relayd will open v4l2sink to v4l2loopback OUTPUT deivce, and it will only open libcamhal based GStreamer source element, and therefore underlying camera hardware, until received new client notifications via V4L2 Event API that is introduced in this SRU.

Besides, frame sizes/intervals enumeration is also fixed to meet better compliance with user apps.

[Test Plan]

v4l2loopback doesn't support V4L2 Event API until recently, so requests for the usage will always fail:

  struct v4l2_event_subscription sub;
  int fd;

  memset (&sub, 0, sizeof (sub));
  sub.type = ...;
  if (ioctl (fd, VIDIOC_SUBSCRIBE_EVENT, &sub) == 0) // fail

With this fix, it shall support Event API operations, e.g. VIDIOC_SUBSCRIBE_EVENT, VIDIOC_UNSUBSCRIBE_EVENT, VIDIOC_DQEVENT.

[Where problems could occur]

While a custom type of V4L2 event is now registered for OEM projects, software expected same ID (numerically equivalent to V4L2_EVENT_PRIVATE_START) may get confused. While this falls in the private usage section, it's not supposed to be used in general, and when it does, it should be under some presumptions, e.g. selected hardware, so it's unlikely to happen on OEM projects.

For generic Ubuntu, programs may begin to take advantage of this new capability to update its UI, or to take other actions after receiving desired events. There might be behavior changes, but should be under the original design if was done carefully.

[Other Info]

For Focal backports, 0.12.5-1 is equivalent to 0.12.3-1ubuntu0.3 plus micro version updates, so there should be little risk backport a new release, and we can drop additional patches carried. This implies bug 1905613.

This focal backport includes changes for bug 1921474, bug 1930208, and bug 1936250, so it will then become synced with Impish and newer again. The 0007-compliance-stop-declaring-V4L2_CAP_VIDEO_M2M-capabil.patch carried in bug 1930208 is deliberately skipped to avoid bug 1946660.

========= original bug report ==========

In addition to kernel/firmware proposed in bug 1921345, several user space daemons/libraries, as well as fixes/features in v4l2loopback-dkms, are also required to have seamless support for legacy/existing Linux kernel V4L2 API based applications to adopt libcamera or Intel libcamhal provided camera interfaces.

The idea is to add a v4l2 streaming relay daemon (currently developed in [1] and packaged in [2]) that helps redirecting V4L2 buffer streams into v4l2loopback output device, and then legacy apps open v4l2loopback capture device in the ordinary way.

  hw -> libcamera/libcamhal -> v4l2-relayd -> v4l2loopback -> v4l2-based apps

To achieve this, we'd like v4l2-relayd to only open GStreamer icamerasrc pipeline (provided by [3],[4]) when there is actual usage request from v4l2-based apps for privacy and power consumption's concerns. However, current (0.12.5 and therefore Ubuntu/Debian 0.12.5-1) doesn't have any available mechanism for this, therefore a V4L2 Event API based proposal has been sent and accepted by upstream[5][6].

This is needed for Ubuntu OEM projects with MIPI cameras through Intel IPU6 (Imaging Processing Unit version 6).

[1]: https://gitlab.com/vicamo/v4l2-relayd
[2]: https://code.launchpad.net/~oem-solutions-engineers/v4l2-relayd/+git/packaging
[3]: https://github.com/intel/ipu6-camera-hal
[4]: https://github.com/intel/icamerasrc
[5]: https://github.com/umlaeute/v4l2loopback/pull/345
[6]: https://github.com/umlaeute/v4l2loopback/pull/352

[You-Sheng Yang (vicamo)]

Attach debdiff built from https://code.launchpad.net/~vicamo/ubuntu/+source/v4l2loopback/+git/v4l2loopback/+ref/bug-1921474/support-client-usage-v4l2-event-api/hirsute.

[You-Sheng Yang (vicamo)]

Attach debdiff built from https://code.launchpad.net/~vicamo/ubuntu/+source/v4l2loopback/+git/v4l2loopback/+ref/bug-1921474/support-client-usage-v4l2-event-api/focal.

[You-Sheng Yang (vicamo)]

PPA build for verification: https://launchpad.net/~vicamo/+archive/ubuntu/ppa-1921474 (focal, hirsute)

[You-Sheng Yang (vicamo)]

Need help from ubuntu-sponsors.

[Launchpad Janitor (janitor)]

This bug was fixed in the package v4l2loopback - 0.12.5-1ubuntu1

---------------
v4l2loopback (0.12.5-1ubuntu1) hirsute; urgency=medium

  * Support client usage notification via V4l2 Event API (LP: #1921474)
    - dev: initialize per opener v4l2_fh
    - event: install event (un)subscribe hook
    - event: support polling for events
    - event: fix backward compatibility to 3.16
    - UBUNTU: SAUCE: track active readers
    - UBUNTU: SAUCE: event: support V4L2_EVENT_PRI_CLIENT_USAGE
    - compliance: stop declaring V4L2_CAP_VIDEO_M2M capability
    - compliance: fix vidioc_enum_frameintervals in output side
    - compliance: fix enum frame sizes/intervals errors
    - compliance: fix "fmtdesc.type was modified" error

 -- You-Sheng Yang <email address hidden> Mon, 29 Mar 2021 17:02:42 +0800

[Timo Aaltonen (tjaalton)]

An upload of v4l2loopback to focal-proposed has been rejected from the upload queue for the following reason: "The version is wrong, newer than what's in hirsute, and don't drop debian/changelog entries when backporting".

[Brian Murray (brian-murray)]

Given that the focal upload for this was rejected from the SRU queue and there hasn't been a new debdiff attached to the bug report I am unsubscribing the sponsors team.

[You-Sheng Yang (vicamo)]

Attach v4l2loopback_0.12.3-1ubuntu0.4_0.12.3-1ubuntu0.5.debdiff

v4l2loopback (0.12.3-1ubuntu0.5) focal; urgency=medium

  * Fix coverity errors. (LP: #1930208)
    - v4l2loopback.c: Fix unchecked return value in vidioc_s_fmt_out()
    - v4l2loopback.c: resource leak in v4l2_loopback_open()
    - v4l2loopback.c: NULL dereference in free_buffers()
    - UBUNTU: SAUCE: coverity: fix unchecked return value
    - UBUNTU: SAUCE: coverity: fix null pointer dereference
    - UBUNTU: SAUCE: coverity: use strlcpy instead of strcpy
    - Don't fail if allocating 0-sized buffers

  * Support client usage notification via V4l2 Event API (LP: #1921474)
    - dev: initialize per opener v4l2_fh
    - event: install event (un)subscribe hook
    - event: support polling for events
    - event: fix backward compatibility to 3.16
    - UBUNTU: SAUCE: track active readers
    - UBUNTU: SAUCE: event: support V4L2_EVENT_PRI_CLIENT_USAGE
    - compliance: fix vidioc_enum_frameintervals in output side
    - compliance: fix enum frame sizes/intervals errors
    - compliance: fix "fmtdesc.type was modified" error

 -- You-Sheng Yang <email address hidden> Wed, 29 Dec 2021 17:29:06 +0800

[You-Sheng Yang (vicamo)]

A prebuilt version is also uploaded to https://launchpad.net/~vicamo/+archive/ubuntu/ppa-1921474

[Robie Basak (racb)]

> On a MIPI camera through Intel IPU6 platform that its raw V4L2 loopback interface is preserved for Intel Camera HAL libraries, a relay daemon + v4l2loopback is used to allow the usage of legacy V4L2 based apps.

Why is this camera's raw V4L2 loopback interface being "preserved for Intel Camera HAL libraries"? Is this a requirement for hardware enablement of this device (and if so, why?), or merely a userspace software rearrangement? What functionality won't work if Ubuntu users on Focal just use the V4L2 interface without using the "Intel Camera HAL libraries"? What package provides the "Intel Camera HAL libraries"?

[You-Sheng Yang (vicamo)]

Hi,

On Wed, Jan 5, 2022 at 8:35 PM Robie Basak <email address hidden> wrote:
>
> > On a MIPI camera through Intel IPU6 platform that its raw V4L2
> loopback interface is preserved for Intel Camera HAL libraries, a relay
> daemon + v4l2loopback is used to allow the usage of legacy V4L2 based
> apps.
>
> Why is this camera's raw V4L2 loopback interface being "preserved for
> Intel Camera HAL libraries"?

The Intel IPU camera provides only raw image access through kernel
V4L2 interface, and one will need the corresponding userspace
middleware to manipulate the image/video stream. For IPU6 that is
being used on a certain oem platforms, following components are
needed:

* https://github.com/intel/ipu6-camera-bins: prebuilt proprietary
binaries and firmware blobs,
* https://github.com/intel/ipu6-camera-hal: camera hardware
abstraction layer library,
* https://github.com/intel/icamerasrc: gstreamer element plugin.

So these components will allow a camera app, with the knowledge of
icamerasrc, to retrieve normal image/video streams. For all existing,
legacy, v4l2-only camera apps, they will not be able to find
/dev/video* as they don't exist at all. And even there are, they will
still need the above components to manipulate the image/video for
correct presentation.

In order to provide best backward compatibility, a v4l2-relayd is used
to open hardware camera via gstreamer icamerasrc element and redirect
the stream back into kernel v4l2loopback device, then a general,
innocent userspace camera app may directly use the video device
provided by v4l2loopback.

All the components can be found in
https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6,
and are being proposed for inclusion to Ubuntu archive at. the same
time

> Is this a requirement for hardware
> enablement of this device (and if so, why?), or merely a userspace
> software rearrangement?

Mostly in the userspace, but we'll need v4l2loopback as a way to
redirect processed video/image streams.

> What functionality won't work if Ubuntu users on
> Focal just use the V4L2 interface without using the "Intel Camera HAL
> libraries"?

This SRU is only for adding V4L2 Event APIs and a few coverity fixes,
so it doesn't really cause functionality changes.

However, it's true that v4l2-relayd now installs a
/etc/modprobe.d/v4l2-relayd.conf file that specifies module parameters
for v4l2loopback. So for Ubuntu users without v4l2-relayd installed,
there is no visible difference.

> What package provides the "Intel Camera HAL libraries"?

Listed above.

[Robie Basak (racb)]

Thank you for the detailed explanation!

> All the components can be found in
https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6,
and are being proposed for inclusion to Ubuntu archive at. the same
time

What's the status of the inclusion of the entire stack in Jammy? I don't see ipu6-camera-hal or ipu6-camera-bins packages in Ubuntu at all, for example.

Right now, it seems to me that you're essentially asking for new features in v4l2loopback in Focal because they are required to make this new out-of-archive proprietary stack work. I assume that you're seeking to justify this SRU on the basis of our hardware enablement policy, as you've not stated any other justification in your SRU supplied information. But I'm not sure how this fits in with Ubuntu's existing hardware enablement policy, unless every required component for the entire enablement is also being added to Focal. Otherwise, if out-of-archive software will still be required to make the hardware enablement work, then why do the required feature additions to v4l2loopback need to go into the archive rather than the out-of-archive place?

Ubuntu SRU policy requires that changes being made are "fixed" in the development release first. In this case I think the entire solution needs to be demonstrated working in Jammy without the need for an out-of-archive source - not just the v4l2loopback component. Otherwise, I don't think it makes sense to justify the feature addition to v4l2loopback on the basis of hardware enablement (as I assume you're seeking to do) as there wouldn't be a hardware enablement in Ubuntu itself actually happening.

Please complete the entire hardware enablement in Jammy first - presumably using the restricted archive component as necessary - before proceeding with the hardware enablement in the Ubuntu archive in Focal. In the meantime, if users need to install out-of-archive (eg. PPA) software to get the stack working anyway, then it shouldn't be onerous for them to also bump v4l2loopback with the feature required.

The reason I'm pushing on this is that I think there's quite a bit of risk in how the rest of the enablement will happen in Focal, which might lead to development iteration for users in a stable release which really should be avoided. We mitigate this by requiring fixes to be demonstrated to be fully functional and tested in the development release first, and I think that it's particularly important to ensure that is done here because this is a particularly complex case. I would also prefer to see the entire enablement in Focal being performed at once (after Jammy is complete) so that it can be tested and verified as a whole, rather than a component at a time. Otherwise we'll end up testing something different from the actual goal for users when doing v4l2loopback first, which risks those further iterations in a stable release. With just the v4l2loopback feature being added here, it becomes impossible for me to add "Check that a user can actually use the hardware being enabled as expected" to the test plan for SRU verification.

[You-Sheng Yang (vicamo)]

@racb, please advice the next action to be taken for this SRU. Shall we suspend the work here and wait for all other bits being ready in Jammy?

[Robie Basak (racb)]

On Thu, Jan 13, 2022 at 07:52:36AM -0000, You-Sheng Yang wrote:
> @racb, please advice the next action to be taken for this SRU. Shall we
> suspend the work here and wait for all other bits being ready in Jammy?

You can continue work on the various pieces, but I don't think it's a
good idea to proceed into the archive without the other pieces being
ready so that everything can be reviewed together. Without seeing
everything else, I'm not able to say whether what you have planned will
be acceptable for SRU.

[Robie Basak (racb)]

An upload of v4l2loopback to focal-proposed has been rejected from the upload queue for the following reason: "Clearing this from the queue for now as it isn't ready for an SRU to Focal. See https://bugs.launchpad.net/ubuntu/+source/v4l2loopback/+bug/1921474/comments/12 and subsequent comments for details.".