Support MIPI camera through Intel IPU6

Bug #1921345 reported by You-Sheng Yang
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
HWE Next
Undecided
Unassigned
linux (Ubuntu)
Status tracked in Jammy
Focal
Undecided
Unassigned
Jammy
Undecided
Unassigned
linux-firmware (Ubuntu)
Undecided
Unassigned
Focal
High
You-Sheng Yang
Hirsute
Undecided
Unassigned
Jammy
Undecided
Unassigned
linux-oem-5.10 (Ubuntu)
Focal
High
You-Sheng Yang
linux-oem-5.13 (Ubuntu)
Status tracked in Jammy
Focal
High
You-Sheng Yang
Jammy
Undecided
Unassigned
linux-oem-5.14 (Ubuntu)
Status tracked in Jammy
Focal
Undecided
You-Sheng Yang
Jammy
Undecided
Unassigned

Bug Description

[SRU Justification]

[Impact]

There are new platforms with MIPI camera based on Intel IPU6(Imaging
Processing Unit version 6) currently unsupported in Ubuntu.

[Fix]

Intel IPU6 can be enabled by integration of following kernel driver,
firmware, userspace HAL libraries as well as a GStreamer source element:

  * https://github.com/intel/ipu6-drivers
  * https://github.com/intel/ipu6-camera-bins
  * https://github.com/intel/ipu6-camera-hal
  * https://github.com/intel/icamerasrc

This patchset took kernel patches from intel/ipu6-drivers.

[Test Case]

With kernel/firmware properly installed, the hardware should be ready for
further development:

  $ dmesg | grep ipu
  intel-ipu6 intel-ipu: enabling device (0000 -> 0002)
  intel-ipu6 intel-ipu: Device 0x9a19 (rev: 0x1)
  intel-ipu6 intel-ipu: physical base address 0x6054000000
  intel-ipu6 intel-ipu: mapped as: 0x0000000097793328
  intel-ipu6 intel-ipu: IPU in secure mode
  intel-ipu6 intel-ipu: IPC reset done
  intel-ipu6 intel-ipu: cpd file name: intel/ipu6_fw.bin
  intel-ipu6 intel-ipu: FW version: 20201222
  intel-ipu6 intel-ipu: Sending BOOT_LOAD to CSE
  intel-ipu6 intel-ipu: Sending AUTHENTICATE_RUN to CSE
  intel-ipu6 intel-ipu: CSE authenticate_run done
  intel-ipu6 intel-ipu: IPU driver version 1.0
  intel-ipu6-psys intel-ipu6-psys0: pkg_dir entry count:8
  intel-ipu6-psys intel-ipu6-psys0: psys probe minor: 0
  intel-ipu6-isys intel-ipu6-isys0: bind ov01a1s 20-0036 nlanes is 1
port is 1
  intel-ipu6-isys intel-ipu6-isys0: All sensor registration completed.
  intel-ipu6-isys intel-ipu6-isys0: stream on ov01a1s 20-0036

There should be nearly 24 video4linux devices created under /dev.

[Where problems could occur]

MIPI camera through Intel IPU6 takes also firmware blobs loaded in
runtime from user space, addtional softwares are also required to
fully enable it for oridinary use. The exposed video devices are hidden
from general users and a relay daemon+v4l2loopback is used to enable
use from existing applications.

>From kernel's point of view, there is still a known issue that it takes
root priviledge to access these devices.

[Other Info]

So far Intel has no plan to commit this driver to upstream yet, so it's
only nominated for oem-5.10. It will only be nominated to generic
kernels when the plan/requirements have changed.

The original driver Kconfig would disable VIDEO_IPU3_CIO2, but that's
reverted to avoid unnecessary changes to existing users.

========== original bug description ==========

The Intel imaging processing unit version 6, found in Intel SoCs and used for capturing images and video from a camera sensor is enabled by integration of following kernel driver, firmware, userspace HAL libraries as well as a GStreamer source element:

  * https://github.com/intel/ipu6-drivers
  * https://github.com/intel/ipu6-camera-bins
  * https://github.com/intel/ipu6-camera-hal
  * https://github.com/intel/icamerasrc

With all components properly installed, the hardware should be ready for further development:

  intel-ipu6 intel-ipu: enabling device (0000 -> 0002)
  intel-ipu6 intel-ipu: Device 0x9a19 (rev: 0x1)
  intel-ipu6 intel-ipu: physical base address 0x6054000000
  intel-ipu6 intel-ipu: mapped as: 0x0000000097793328
  intel-ipu6 intel-ipu: IPU in secure mode
  intel-ipu6 intel-ipu: IPC reset done
  intel-ipu6 intel-ipu: cpd file name: intel/ipu6_fw.bin
  intel-ipu6 intel-ipu: FW version: 20201222
  intel-ipu6 intel-ipu: Sending BOOT_LOAD to CSE
  intel-ipu6 intel-ipu: Sending AUTHENTICATE_RUN to CSE
  intel-ipu6 intel-ipu: CSE authenticate_run done
  intel-ipu6 intel-ipu: IPU driver version 1.0
  intel-ipu6-psys intel-ipu6-psys0: pkg_dir entry count:8
  intel-ipu6-psys intel-ipu6-psys0: psys probe minor: 0
  intel-ipu6-isys intel-ipu6-isys0: bind ov01a1s 20-0036 nlanes is 1 port is 1
  intel-ipu6-isys intel-ipu6-isys0: All sensor registration completed.
  intel-ipu6-isys intel-ipu6-isys0: stream on ov01a1s 20-0036

This one equips an OV01A1S sensor, and HM11B1 is also supported.

---
ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: u 1721 F.... pulseaudio
CasperMD5CheckResult: skip
Dependencies:

DistributionChannelDescriptor:
 # This is the distribution channel descriptor for the OEM CDs
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-focal-amd64-20200502-85+fossa-mewtwo+X74
DistroRelease: Ubuntu 20.04
InstallationDate: Installed on 2021-03-25 (0 days ago)
InstallationMedia: Ubuntu 20.04 "Focal" - Build amd64 LIVE Binary 20200502-05:58
MachineType: Dell Inc. Latitude 9420
Package: linux-oem-5.10
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.10.0-1019-oem root=UUID=f7f15afa-43c6-46b8-b917-89735875cb39 ro
ProcVersionSignature: Ubuntu 5.10.0-1019.20-oem 5.10.18
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-5.10.0-1019-oem N/A
 linux-backports-modules-5.10.0-1019-oem N/A
 linux-firmware 1.187.10+lp1921345.1.intel.ipu6
Tags: focal third-party-packages
Uname: Linux 5.10.0-1019-oem x86_64
UnreportableReason: This is not an official Ubuntu package. Please remove any third party package and try again.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 01/25/2021
dmi.bios.release: 89.5
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 89.5.23
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 10
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr89.5.23:bd01/25/2021:br89.5:svnDellInc.:pnLatitude9420:pvr:rvnDellInc.:rn:rvr:cvnDellInc.:ct10:cvr:
dmi.product.family: Latitude
dmi.product.name: Latitude 9420
dmi.product.sku: 0A32
dmi.sys.vendor: Dell Inc.

CVE References

You-Sheng Yang (vicamo)
Changed in linux-oem-5.10 (Ubuntu Hirsute):
status: New → Invalid
Changed in linux-firmware (Ubuntu Hirsute):
status: New → Invalid
Changed in linux-oem-5.10 (Ubuntu Focal):
status: New → In Progress
importance: Undecided → High
assignee: nobody → You-Sheng Yang (vicamo)
Changed in linux-firmware (Ubuntu Focal):
status: New → In Progress
importance: Undecided → High
assignee: nobody → You-Sheng Yang (vicamo)
tags: added: oem-priority originate-from-1917691 somerville
Revision history for this message
You-Sheng Yang (vicamo) wrote :
Revision history for this message
You-Sheng Yang (vicamo) wrote : AlsaInfo.txt

apport information

tags: added: apport-collected focal third-party-packages
description: updated
Revision history for this message
You-Sheng Yang (vicamo) wrote : CRDA.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : CurrentDmesg.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : IwConfig.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : Lspci.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : Lspci-vt.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : Lsusb.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : Lsusb-t.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : Lsusb-v.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : ProcInterrupts.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : ProcModules.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : RfKill.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : UdevDb.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : WifiSyslog.txt

apport information

Revision history for this message
You-Sheng Yang (vicamo) wrote : acpidump.txt

apport information

summary: - Support Intel IPU6 camera
+ Support MIPI camera through Intel IPU6
description: updated
You-Sheng Yang (vicamo)
description: updated
Revision history for this message
You-Sheng Yang (vicamo) wrote : Re: [Bug 1921345] Re: Support Intel IPU6 camera

Attach logs captured on 202101-28643 (borrowed from Dell), BIOS
89.5.23 01/25/2021, image
dell-bto-focal-fossa-mewtwo-X74-20210324-5.iso.

Basically it works out-of-box for once, 1280x720@30fps, then Intel
icamerasrc element doesn't release fd upon pipeline stop, leaving
error messages in system journal:

  v4l2-relayd-wrapper[898]: 03-25 08:03:49.421: [DBG]:
CamHAL_CameraHal:@init, mInitTimes:2, return without running
  v4l2-relayd-wrapper[898]: 03-25 08:03:49.421: [DBG]:
CamHAL_CameraHal:@deviceOpen: open multi times

Btw, hack for cameraNumber in /etc/camera/libcamhal_profile.xml is no
longer necessary.

Revision history for this message
You-Sheng Yang (vicamo) wrote :
description: updated
Revision history for this message
You-Sheng Yang (vicamo) wrote :
Rex Tsai (chihchun)
tags: added: ipu6
Revision history for this message
You-Sheng Yang (vicamo) wrote :

SRU: https://lists.ubuntu.com/archives/kernel-team/2021-April/118829.html (oem-5.10, v2 to include Intel WW14 release)

Revision history for this message
You-Sheng Yang (vicamo) wrote :

SRU: https://lists.ubuntu.com/archives/kernel-team/2021-April/119220.html (oem-5.10, v3 to include Intel 20210413 release)

tags: added: originate-from-1922830
Timo Aaltonen (tjaalton)
Changed in linux-oem-5.10 (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello You-Sheng, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in linux-firmware (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
You-Sheng Yang (vicamo) wrote :

Verified linux-oem-5.10 kernel packages version 5.10.0-1024.25 from focal-proposed.

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-firmware - 1.187.12

---------------
linux-firmware (1.187.12) focal; urgency=medium

  * Support MIPI camera through Intel IPU6 (LP: #1921345)
    - SAUCE: intel-ipu6: Add IPU6 firmware files
    - SAUCE: intel-ipu6: update IPU6 Release_20210121 WW04 firmware files

  * Add realtek 8852 bluetooth fimware (LP: #1924203
    - rtl_bt: Add firmware and config files for RTL8852A BT USB chip

 -- Seth Forshee <email address hidden> Mon, 26 Apr 2021 07:58:04 -0500

Changed in linux-firmware (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Tiffany (hsiaoting)
information type: Public → Public Security
information type: Public Security → Public
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-oem-5.10 - 5.10.0-1025.26

---------------
linux-oem-5.10 (5.10.0-1025.26) focal; urgency=medium

  * focal/linux-oem-5.10: 5.10.0-1025.26 -proposed tracker (LP: #1926155)

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * CirrusLogic: Cracking noises appears in built-in speaker when output volume
    is set >80% (LP: #1924997)
    - SAUCE: ALSA: hda/cirrus: Use CS8409 Equalizer to fix abnormal sounds on
      Bullseye

 -- Timo Aaltonen <email address hidden> Mon, 26 Apr 2021 16:18:11 +0300

Changed in linux-oem-5.10 (Ubuntu Focal):
status: Fix Committed → Fix Released
Mathew Hodson (mhodson)
no longer affects: linux-oem-5.10 (Ubuntu)
no longer affects: linux-oem-5.10 (Ubuntu Hirsute)
Mathew Hodson (mhodson)
Changed in linux-firmware (Ubuntu):
status: Invalid → Won't Fix
Changed in linux-firmware (Ubuntu Hirsute):
status: Invalid → Won't Fix
You-Sheng Yang (vicamo)
Changed in linux-oem-5.13 (Ubuntu):
status: New → Invalid
Changed in linux-oem-5.13 (Ubuntu Focal):
status: New → Triaged
assignee: nobody → You-Sheng Yang (vicamo)
importance: Undecided → High
You-Sheng Yang (vicamo)
Changed in linux-oem-5.13 (Ubuntu Focal):
status: Triaged → In Progress
Revision history for this message
You-Sheng Yang (vicamo) wrote :

For oem-5.13, draft fixes and prebuilt kernel available in same PPA starting from version 5.13.0-9009.10+staging.4 . Need some more polishment before landing.

Revision history for this message
You-Sheng Yang (vicamo) wrote :
Revision history for this message
Marcello Sylvester Bauer (sylv-io) wrote :

> From kernel's point of view, there is still a known issue that it takes
> root priviledge to access these devices.

There is a way to run the gstreamer plugin as normal user via udev rule:
https://github.com/intel/icamerasrc/pull/9

The IPU6 processing subsystem does not use a V4l2 device node in contrast to the Input system.
Therefore it is required to fix the access permission to `/dev/ipu-psys0`.

Timo Aaltonen (tjaalton)
Changed in linux-oem-5.13 (Ubuntu Focal):
status: In Progress → Fix Committed
Revision history for this message
You-Sheng Yang (vicamo) wrote :

Need to fix API change introduced in commit 0d346d2a6f54 ("media: v4l2-subdev: add subdev-wide state struct") when compiled against v5.14 and on.

Revision history for this message
You-Sheng Yang (vicamo) wrote : Re: [Bug 1921345] Re: Support MIPI camera through Intel IPU6

On oem-5.14, we have built a test kernel in
https://launchpad.net/~vicamo/+archive/ubuntu/linux-staging for
initial taste for internal use. Will copy to
https://launchpad.net/~oem-solutions-group/+archive/ubuntu/intel-ipu6
when ready.

So far we find there might be intermittent failure when probing either sensor.

For ov01a1s sku, cold boot always fails to probe the sensor, but warm
boot works fine.

For hm11b1 sku, oem-5.14 kernel, cold boot always fails, and there are
50% chances that it may fail probing in warm boots. And this can also
be found with oem-5.13 kernel with a slightly different symptom that
cold boot always succeeds, and 50% failures in warm boots.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-oem-5.13 - 5.13.0-1012.16

---------------
linux-oem-5.13 (5.13.0-1012.16) focal; urgency=medium

  * focal/linux-oem-5.13: 5.13.0-1012.16 -proposed tracker (LP: #1942861)

  * Fix sluggish r8169 by disaling ASPM L1.2 (LP: #1942830)
    - SAUCE: r8169: Disable ASPM L1.2

  * CVE-2021-3609
    - can: bcm: delay release of struct bcm_op after synchronize_rcu()
    - can: j1939: j1939_sk_init(): set SOCK_RCU_FREE to call sk_destruct() after
      RCU is done

 -- Timo Aaltonen <email address hidden> Tue, 07 Sep 2021 11:49:54 +0300

Changed in linux-oem-5.13 (Ubuntu Focal):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu Focal):
status: New → Won't Fix
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
Timo Aaltonen (tjaalton)
Changed in linux-oem-5.14 (Ubuntu Jammy):
status: New → Invalid
Timo Aaltonen (tjaalton)
Changed in linux-oem-5.14 (Ubuntu Focal):
status: New → In Progress
assignee: nobody → You-Sheng Yang (vicamo)
Revision history for this message
You-Sheng Yang (vicamo) wrote :

Move oem-5.14, jammy porting to bug 1955383.

Changed in linux (Ubuntu Jammy):
status: Confirmed → Invalid
Changed in linux-oem-5.14 (Ubuntu Focal):
status: In Progress → Invalid
You-Sheng Yang (vicamo)
Changed in hwe-next:
status: New → Fix Released
Revision history for this message
Krister Swenson (thekswenson) wrote :

Hi folks, thanks for all your great work!

I'm running 21.10 on my Dell 7320 Detachable.
I'm wondering if you could tell me what the easiest way is to get my IPU6 tigerlake webcam to work?

To post a comment you must log in.