Update the ModemManager to 1.16.6-2 to support some modems in Focal and Hirsute releases.

[Jerry Lee]

[Impact]

Some IOT products use wireless modems which can be working only when recent versions of the ModemManager suite are used.

The following 2 modems need the ModemManager suite to be upgraded:
* Foxconn SDX55 T99W175 5G sub6 PCIE Modem
* Quectel SDX24 EM160R-GL 4G LTE CAT16 PCIE Modem

The main fix requested is to add the FCC unlock mechanism for Foxconn modems.
* FCC unlock operation for Foxconn modems
https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/534/commits
* dms: new 'Foxconn Set FCC authentication' command
https://gitlab.freedesktop.org/mobile-broadband/libqmi/-/merge_requests/254/commits

The minimum versions of the ModemManager suite required to enable the support for the above 2 mentioned modems have been verified: (LP: #1928665)
* ModemManager: 1.16.6
* libmbim: 1.24.8
* libqmi : 1.28.6

The ModemManager suite in the Impish release meets the requirements.

[Test Plan]

= How to Reproduce the Bug =

Execute the following commands to list the modems detected by ModemManager in Hirsute and Focal releases:

$ mmcli --list-modems
No modems were found

= Test Procedure =

1. Install the Ubuntu system on the tested hardware

  The following images will be used to verify the version of the ModemManager suite:
  * Impish: https://cdimage.ubuntu.com/daily-live/current/impish-desktop-amd64.iso
  * Hirsute: https://releases.ubuntu.com/21.04/ubuntu-21.04-desktop-amd64.iso
  * Focal: https://releases.ubuntu.com/focal/ubuntu-20.04.2.0-desktop-amd64.iso

2. Upgrade the kernel and driver ( for Foxconn and Quectel modem )

  The kernel needs to get some patches from 5.13 and includes a back ported Quectel driver to support these 2 modems.
  We have prepared a kernel packages for testing: https://people.canonical.com/~mschiu77/lp1928665/v2/

3. Install the ModemManager suite ( for Hirsute and Focal releases )

  The ModemManager suite will be installed from the -proposed component:

$ sudo apt update
$ sudo apt install modemmanager
$ sudo apt install libqmi-utils

4. Execute the following commands

4.1 Get the run-time environment

$ uname -ar
$ lsb_release -a
$ mmcli -V
$ qmicli -V

4.2 Check the status of the ModemManager service

$ sudo systemctl status ModemManager.service

4.3 List the detected modems

$ mmcli --list-modems

4.4 Check the modem’s status

$ mmcli --modem 0

4.5 Install and execute Lenovo’s FCC unlock app ( for Quectel modem only )

$ sudo snap install --devmode --dangerous dpr-wwan_1.0-wwan-test_amd64.snap

4.6 Enable the detected modem

$ sudo mmcli --modem 0 --enable

4.7 Check the modem’s status

$ mmcli --modem 0

= Analyze the Tested Result =

1. Check if installed packages are working

  The result of test procedure 4.1 and 4.2 can be used to make sure the installed packages are working.
  If the Modemmanager.service is active(running), the packages are working.

2. Check if the supported modem can be detected

  The result of test procedure 4.3 can be used to see if the modem can be detected by ModemManager or not.
  The supported modems should be listed.

3. Check if the modem can be enabled

  If the modem can be enabled, the state of the modem in the test procedure 4.7 will be set to be registered.

= Certification Validation =

Additionally to the aforementioned test cases, the Certification Team will perform some coverage testing across supported devices to make sure the other modems still work as expected.

[Where problems could occur]

There is a risk that modems supported in the old versions of ModemManager suite may not be supported in the newer versions.

[Other Info]

We need to upgrade to these 3 packages (in Impish) at the same time:
* ModemManager: 1.16.6
* libmbim: 1.24.8
* libqmi : 1.28.6

To support the mentioned 2 modems, the system needs to use kernels which include specific patches and kernel config options .
For ex., there is a patched 5.11 kernel which is working for: https://bugs.launchpad.net/austin/+bug/1928665/comments/7

[Jerry Lee (jerry-lee-tpe)]

The test report is attached as "TestReports-1934286.pdf".

[Jerry Lee (jerry-lee-tpe)]

Attach the Debdiff file for Hirsute(21.04)

[Jerry Lee (jerry-lee-tpe)]

Attach the Debdiff file for Focal(20.04)

The debian package modemmanager 1.16.6-2 can be found in the link: https://launchpad.net/ubuntu/impish/+package/modemmanager

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "modemmanager_1-1.16.6-2-hirsute.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Jerry Lee (jerry-lee-tpe)]

The debian package modemmanager 1.16.6-2 can be found in the link: https://launchpad.net/ubuntu/impish/+package/modemmanager

Attach the Debdiff file for Focal(20.04)

[Jerry Lee (jerry-lee-tpe)]

Attach the Debdiff file for Hirsute (21.04)

[Łukasz Zemczak (sil2100)]

Thank you for providing the debdiffs! From what I see there are no series-specific changes, right? So those are essentially straight backports to hirsute and focal? Anyway, I treated those as such and sponsored both (with slight changelog adjustments) - they are now in the hirsute and focal Unapproved queues, waiting for SRU review.

[Łukasz Zemczak (sil2100)]

hm, so I sponsored the modemmanager bits, but seeing the description we still need to get libqmi from impish into hirsute and focal + libmbim from hirsute to focal, right? Should I prepare backports of those as well?

[Jerry Lee (jerry-lee-tpe)]

Yes:
* there are no series-specific changes. (just one line in the change log from Impish)
* those are essentially straight backports to hirsute and focal

I've corrected the libmbim version to be the same as in Impish.

Thanks a lot!

[Łukasz Zemczak (sil2100)]

Ok, thanks for the info! I uploaded both libqmi and libmbim to both hirsute and focal queues. Now it needs someone to review the SRUs and accept them (I can poke around).

I'll also modify the description to mention the additional testing that the Certification Team will perform on the packages as it will really make the SRU look safer. The more tests performed, the better.

Removing ubuntu-sponsors.

[Brian Murray (brian-murray)]

I'm modifying the description to indicate that the packages should be installed from -proposed on Focal and Hirsute and not from impish (which is what it currently says).

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted modemmanager into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/modemmanager/1.16.6-2~21.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted libmbim into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libmbim/1.24.8-1~21.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted libqmi into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libqmi/1.28.6-1~21.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted modemmanager into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/modemmanager/1.16.6-2~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Looking at the changes in the libmbim upload for Focal I noticed that mbim-proxy was moved from /usr/lib/libmbim/ to /usr/libexec/. This ends up moving a binary that is not included in $PATH and is installed by default on Ubuntu 20.04 LTS afaict. I don't know how many people would have scripts expecting libmbim to be in /usr/lib/libmbim/ but I don't think changing it out from under them is great. Could we patch the package so it exists in both places?

libmbim (1.22.0-2.2) unstable; urgency=medium

  * Non-maintainer upload.
  * Bump debhelper to 12, no further changes
  * Bump Standards-Version to 4.5.0 (no further changes)
  * Move mbim-proxy to /usr/libexec (Closes: #954145)
...

[Jerry Lee (jerry-lee-tpe)]

As the ModemManager is only considered, mbim-proxy is started by the ModemManager service:

"Jul 09 08:20:21 ubuntu-ThinkEdge-SE30 ModemManager[578]: spawning new mbim-proxy (try 1)..."

And, the execution path is decided at https://github.com/linux-mobile-broadband/libmbim/blob/master/src/libmbim-glib/mbim-device.c#L1421:

       argc[0] = g_strdup (LIBEXEC_PATH "/mbim-proxy");

This looks like that the mbim-proxy is only used by the ModemManager service.

[Brian Murray (brian-murray)]

A user still might have written a script or a service which calls /usr/lib/libmbim/mbim-proxy and by moving the binary we would end up introducing a regression and that's a scenario we should consider.

[Jerry Lee (jerry-lee-tpe)]

Hi Brian,

This is a debdiff for libmbim applicable to Focal(1.22.0-2).

The maintainer scripts (libmbim-proxy.postinst, libmbim-proxy.postrm) are added to keep mbim-proxy in both places.

I built this in pbuilder and it builds successfully, and I installed it, the patch works as intended.

[Łukasz Zemczak (sil2100)]

Hello Brian, Jerry!

I have uploaded a new upload of libmbim to the Unapproved queue. Actually a postrm/postinst is not quite the way to go as there are better ways such a task can be realized - so I added a libmbim-proxy.links file to make sure we also create a symlink for the old expected path. This seems to work on my test build here:
https://launchpad.net/~sil2100/+archive/ubuntu/testing/+build/21781947

Brian, could you take a look now? I think this delta is really small and with the additional symlink the regression risk is much much lower.

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted libmbim into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libmbim/1.24.8-1~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

When reviewing libqmi I noticed the same type of change, moving qmi-proxy, so I went ahead and reuploaded the package but with a symlink like Lukasz did for libmbim.

[Brian Murray (brian-murray)]

Hello Jerry, or anyone else affected,

Accepted libqmi into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libqmi/1.28.6-1~20.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Jerry Lee (jerry-lee-tpe)]

Hi Brian,

I can't find the modemmanage from the focal-proposed channel to be upgraded.
https://people.canonical.com/~ubuntu-archive/pending-sru.html shows it is in "Dependency wait".

Please help to check it.

[Aleksander Morgado (aleksander-m)]

And where is the dpr-wwan_1.0-wwan-test_amd64.snap file?

[Jerry Lee (jerry-lee-tpe)]

The snap "dpr-wwan_1.0-wwan-test_amd64.snap" is provided by Lenovo (not published yet), it will be published in the Snap store for Lenovo's system to use it. (it recognizes Lenovo' system then do the FCC unlocking)

[Brian Murray (brian-murray)]

The packages should be available now.

[Aleksander Morgado (aleksander-m)]

> The snap "dpr-wwan_1.0-wwan-test_amd64.snap" is provided by Lenovo (not published yet)

Is there any estimated date when it will be released?

[Kent Lin (kent-jclin)]

Test pass on https://certification.canonical.com/hardware/202010-28320/submission/223095/

[Gabriel Zhi Chen (gabrielzchen)]

The WWAN test is passed on Lenovo ThinkPad L460

https://certification.canonical.com/hardware/201601-20490/submission/223615/

https://certification.canonical.com/hardware/201601-20490/submission/223629/

[Kent Lin (kent-jclin)]

Test Pass on
WWAN Test: https://certification.canonical.com/hardware/202010-28324/submission/223136/
ACPI Stress Test: https://certification.canonical.com/hardware/202010-28324/submission/223167/

[Gene Li (genelicc)]

Test pass with Dell DW5820e [413C:81D9]
WWAN Test: https://certification.canonical.com/hardware/202103-28860/submission/224158/
Stress Test: https://certification.canonical.com/hardware/202103-28860/submission/224158/

[Jerry Lee (jerry-lee-tpe)]

We have verified the packages on focal-proposed:
* The version of the packages tested:
   ModemManager: 1.16.6
   libmbim: 1.24.8
   libqmi: 1.28.6

* The Mobile broadband/WWAN test cases in checkbox have been performed on the packages.
  There are 6 different modems tested and the reports can be downloaded:
  (1) https://certification.canonical.com/hardware/202010-28320/submission/223095/
  (2) https://certification.canonical.com/hardware/201601-20490/submission/223615/
  (3) https://certification.canonical.com/hardware/201601-20490/submission/223629/
  (4) https://certification.canonical.com/hardware/202010-28324/submission/223136/
  (5) https://certification.canonical.com/hardware/202103-28860/submission/224158/
  (6) https://certification.canonical.com/hardware/202103-28860/submission/224158/

* There is a stress test plan performed on the system installing packaes:
  (1) https://certification.canonical.com/hardware/202010-28324/submission/223167/

[Łukasz Zemczak (sil2100)]

Could we also get the hirsute counterparts verified? Thanks!

[Jerry Lee (jerry-lee-tpe)]

We are arranging resource to do the coverage test for hirsute-proposed.

[Łukasz Zemczak (sil2100)]

If we want this in 20.04.3, I would suggest getting the hirsute validation done ASAP! ;)

[Jerry Lee (jerry-lee-tpe)]

The resource is tight so that the plan is to complete the Hirsute verification before Aug. 23.

[Gene Li (genelicc)]

Test pass with Dell DW5820e [413C:81D9] on hirsute

* The version of the packages tested:
   ModemManager: 1.16.6
   libmbim: 1.24.8
   libqmi: 1.28.6

[Kent Lin (kent-jclin)]

Test pass with Dell DW5821e [413C:81D7] on hirsute

* The version of the packages tested:
   ModemManager: 1.16.6
   libmbim: 1.24.8
   libqmi: 1.28.6

[Jerry Lee (jerry-lee-tpe)]

We have verified the packages on hirsute-proposed:
* The version of the packages tested:
   ModemManager: 1.16.6
   libmbim: 1.24.8
   libqmi: 1.28.6

* There are 2 different modems tested and passed the testing:
  (1) DW5820e [413C:81D9]
  (2) DW5821e [413C:81D7]

[Launchpad Janitor (janitor)]

This bug was fixed in the package modemmanager - 1.16.6-2~21.04

---------------
modemmanager (1.16.6-2~21.04) hirsute; urgency=medium

  * Backport to hirsute. (LP: #1934286)

 -- Jerry Lee <email address hidden> Wed, 07 Jul 2021 11:55:02 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package libmbim - 1.24.8-1~21.04

---------------
libmbim (1.24.8-1~21.04) hirsute; urgency=medium

  [ Jerry Lee ]
  * Backport to hirsute. (LP: #1934286)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 08 Jul 2021 14:21:47 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package libqmi - 1.28.6-1~21.04

---------------
libqmi (1.28.6-1~21.04) hirsute; urgency=medium

  [ Jerry Lee ]
  * Backport to hirsute. (LP: #1934286)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 08 Jul 2021 14:19:18 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for modemmanager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package modemmanager - 1.16.6-2~20.04

---------------
modemmanager (1.16.6-2~20.04) focal; urgency=medium

  * Backport to focal. (LP: #1934286)

 -- Jerry Lee <email address hidden> Wed, 07 Jul 2021 10:10:02 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package libmbim - 1.24.8-1~20.04

---------------
libmbim (1.24.8-1~20.04) focal; urgency=medium

  [ Łukasz 'sil2100' Zemczak ]
  * Add debian/libmbim-proxy.links to make sure the mbim-proxy binary is in
    both the old and new path.

  [ Jerry Lee ]
  * Backport to focal. (LP: #1934286)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Wed, 14 Jul 2021 10:30:15 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package libqmi - 1.28.6-1~20.04

---------------
libqmi (1.28.6-1~20.04) focal; urgency=medium

  [ Jerry Lee ]
  * Backport to focal. (LP: #1934286)

  [ Brian Murray ]
  * Add debian/libqmi-proxy.links so that qmi-proxy is in both the old and new
    path.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 08 Jul 2021 14:19:18 +0200

[Jerry Lee (jerry-lee-tpe)]

BTW, the Lenovo's FCC unlock snap is ready for testing.

https://snapcraft.io/lenovo-wwan-dpr

[Pirouette Cacahuète (lissyx)]

Sadly, Quectel EM120 is not supported, so I can't use mine on ThinkPad P14s Gen2 laptop.

[Pirouette Cacahuète (lissyx)]

Looks like https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/402#note_1064882 does help, I have been able to enable my EM120 on 21.10. Take care to change the proper defines in the provided C code. The `mbim2sar.so` comes from the Snap package linked above. Thanks to Joar Wandborg !