Support Quectel's modem(EM120/160) firmware update, fwupd

[Jason Yen]

[Impact]

 * EM120/160 firmware update's fw can't get updated.

[Test Plan]

 * Use a machine with EM120/160 and test fw upgrade or reinstall.

For the rest of the SRU document, please check lp:1982103

Note: the SRU of fwupd is following the exception in
 https://wiki.ubuntu.com/firmware-updates. We also need to
 upgrade libxmlb due to fwupd is its only major consumer,
 and without that, we could not have the correct fwupd function
 on this one.
  --

The current fwupd packages in focal have to be rebuild with the latest 1.18.6 modem manager[1] in updates channel.

In the modem-manager plugin of fwupd, there are several version check macros to determine the version of modem-manager, for instance:

https://git.launchpad.net/ubuntu/+source/fwupd/tree/plugins/modem-manager/fu-mm-device.c?h=ubuntu/focal-updates#n22

    #if MM_CHECK_VERSION(1, 17, 2)
    #include "fu-firehose-updater.h"
    #endif

If the version is equal to or newer than 1.17.2, fwupd will have different behaviors. And the previous version in Focal is 1.16.6 so we have to rebuild or do a version bump to make sure it compile with the new modem manager.

[1]
https://launchpad.net/ubuntu/+source/modemmanager/1.18.6-1~ubuntu20.04.1

[Yuan-Chen Cheng (ycheng-twn)]

AI: 1. make sure if jammy is also affected. 2. prepare debdiff.

[Richard Hughes (richard-hughes)]

If it helps, 1.8.2 is coming in a few days...

[Yuan-Chen Cheng (ycheng-twn)]

Thanks Richard. We need this for focal (22.04). To reduce risk, it's more likely we'll just do a kind of no-change-rebuild on focal.

[Yuan-Chen Cheng (ycheng-twn)]

Per test on jammy / fwupd 1.7.5, I can see "EM160R GL" in the
output of fwupdmgr get-devices.

So we only need to fix this issue in focal.

[Yuan-Chen Cheng (ycheng-twn)]

Create a ppa with the test deb and it works from my side as I enable lvfs-testing remotes.

https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-focal-mm1

Wait verification from another person and plan to start SRU process after that.

[Yuan-Chen Cheng (ycheng-twn)]

Per test from another gentleman, also need to upgrade libxmlb in focal to version >= 0.2.2

focal: libxmlb1 0.1.15-2
jammy: libxmlb2 0.3.6-2build1
kinetic: libxmlb2 0.3.8-1

Ref: https://github.com/fwupd/fwupd/blob/1.7.5/plugins/modem-manager/fu-firehose-updater.c#L723

Logs:

In this case, l.0.1.15 generates:

  Jul 20 14:00:57 ubuntu fwupd[2207]: 06:00:57:0982 FuPluginMm running command 'erase'...

  Jul 20 14:00:57 ubuntu fwupd[2207]: 06:00:57:0982 FuPluginMm writing: <?xml version="1.0" encoding="UTF-8" ?>.<data>.<erase a_rawdata_start_sector="6528" b_rawdata_num_sector="640" c_project_name="EM160RGL" d_project_type="STD" vendor="quectel"></erase></data>

  Jul 20 14:00:57 ubuntu fwupd[2207]: 06:00:57:0989 FuPluginMm reading: <?xml version="1.0" encoding="UTF-8" ?>.<data>.<log value="ERROR: Failed to parse xml, error at offset 190, 190" /></data>

But 0.2.2 generates different and correct:

  <erase a_rawdata_start_sector="6528" b_rawdata_num_sector="640" c_project_name="EM160RGL" d_project_type="STD" vendor="quectel" />

[Yuan-Chen Cheng (ycheng-twn)]

updated focal/libxmlb and fwupd in https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-focal-mm1

Please test again to see if it could properly update the firmware of EM120/160.

[Mario Limonciello (superm1)]

Suggest to redo it to 1.8.3 in your PPA. There is a bunch of other bugs that might want to take that (look at focal targeted) so may want fo just get it all done in one swoop.

[Yuan-Chen Cheng (ycheng-twn)]

If we take the risk of regression into consideration, would you still suggest that we SRU to 1.8.3 in focal?

[Mario Limonciello (superm1)]

Yeah; 1.7.9 would be a safer candidate for focal. It picks up the hwids fix as well as dynamicuser=1 fix.

[Yuan-Chen Cheng (ycheng-twn)]

AI:

ppa for jammy, version 1.8.3
ppa for focal, version 1.7.9

[Yuan-Chen Cheng (ycheng-twn)]

Jammy 1.8.3 ppa: https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-jammy-2
Focal 1.7.9 + libxmlb 0.3.6 ppa: https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-2

[Yuan-Chen Cheng (ycheng-twn)]

Please don't upload to the queue, yet. I got a said-to-be trivial last minutes change request. Please kindly hold for a day or two.

[Yuan-Chen Cheng (ycheng-twn)]

Quectel has verified the new packages in https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-2, it works as expected.

[Yuan-Chen Cheng (ycheng-twn)]

As we are going to upload libxmlb 0.3.6-2build1~20.04.1.

Per quick check, gnome-firmware (universe) also depend on libxmlb1 (libxmlb 0.1.15-2ubuntu1~20.04.1).

One risk is that, if we upgrade libxmlb, it will be a transition of libxmlb1 to libxmlb2.

It seems we will also need to re-build gnome-firmware.

[Yuan-Chen Cheng (ycheng-twn)]

Package depends on libxmlb1:

gnome-software (main)
gnome-firmware (universe)
gnome-software-plugin-flatpak (universe, also build from gnome-software source)

[Yuan-Chen Cheng (ycheng-twn)]

@Mario, I do a not change rebuild for gnome-software and gnome-firmware.

All those live in https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-5

For jammy, updated changelog and put in https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-jammy-4

Could you kindly review and if it's all good, also upload those to Jammy and Focal queue?

[Mario Limonciello (superm1)]

YC - here's my feedback:

Focal
-----
1) I think this bug report will need tasks and test cases for focal for G-S and G-F considering the rebuild.
2) I expect #1966364 is to be fixed as well, you should add to changelog
3) Please double check if #1969976 happens. I think it should be fixed already, but if it's not then there is a slight packaging change that should cherry-pick and I can help you identify it for 1.7.9.
4) Isn't #1982103 fixed for Focal too?
https://github.com/fwupd/fwupd/commit/b2aaddabe456e1a19bc21fd531b84c170507aba5 is part of 1.7.9

Jammy
-----
Some minor typos in changelog. Please fix those up and I'll check it one more time and sponsor it.

[Yuan-Chen Cheng (ycheng-twn)]

@Mario,

Focal: G-S and G-F bug and test case: lp:1984840
All other changelog and debian/control,postinstall change
 shell already live in https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-7

Jammy: found one typo and fixed it in https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-jammy-5

Please kindly review again, thank you.

[Mario Limonciello (superm1)]

1.8.3 (jammy)
-----
My apologies but for Jammy, I think the changelog will need a few more things. These are both expected from my perspective, but they may be jarring/surprising to users and should be documented explicitly on the backport.

1. There was a jump up on the plugin ABI (libfwupdplugin5 -> libfwupdplugin7). That is going to mean a NEW package for an archive administrator to process.
2. The logitech plugin is now activated. This was intentional due to https://bugs.launchpad.net/ubuntu/+source/protobuf-c/+bug/1956617 . This is going to mean that protobuf-c is now promoted from universe to main. This happened in kinetic, but will also need to happen in Jammy now.

Can you document them in the changelog too?

1.7.9 (focal)
-----
To get the package order right with the rebuild, please make sure:
* fwupd has an explicit build-depends on the correct new libxmlb version in debian/control
* gnome-firmware has an explicit build-depends on the correct new fwupd version in debian/control
* gnome-software has a explicit build-depends on the correct new fwupd version in debian/control

Then that should mean libxmlb builds first, then fwupd, then g-s/g-f.

[Yuan-Chen Cheng (ycheng-twn)]

Jammy changelog update: done
Live in: https://launchpad.net/~ycheng-twn/+archive/ubuntu/fwupd-jammy-6

Focal:
* fwupd: already build-dep on libxmlb-dev (>= 0.3.6)
* G-F and G-S: build-dep on new fwupd and libxmlb
Live in: https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-9

[Mario Limonciello (superm1)]

I've sponsored fwupd for Jammy for you and libxmlb in focal for you. They will be pending an archive admin to ack them.

However in focal, I've noticed that you've not updated debian/*.symbols for the changes. Can you please update that and then I'll sponsor the remaining 3 packages.

[Yuan-Chen Cheng (ycheng-twn)]

update focal symbols file and uploaded to https://launchpad.net/~ycheng-twn/+archive/ubuntu/focal-fwupd179-10

[Łukasz Zemczak (sil2100)]

Hello Jason, or anyone else affected,

Accepted libxmlb into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libxmlb/0.3.6-2build1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Jason, or anyone else affected,

Accepted gnome-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnome-firmware/3.34.0-2ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Jason, or anyone else affected,

Accepted gnome-software into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/gnome-software/3.36.1-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Jason, or anyone else affected,

Accepted fwupd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/fwupd/1.7.9-1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted fwupd (1.7.9-1~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

fwupd/1.7.9-1~20.04.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#fwupd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Yuan-Chen Cheng (ycheng-twn)]

with fwupd 1.7.5-3~20.04.1, and `fwupdmgr enable-remote lvfs-testing`.

Can't see EM160R GL.

After install fwupd 1.7.9-1~20.04.1 from the focal-proposed channel,

Can see:

> ├─EM160R GL:
> │ Device ID: 60fd634d16e219aa0a142e9f00e9989654f48b60
> │ Summary: Mobile broadband device
> │ Current version: EM160RGLAPR02A07M4G_15.015.15.015

Given so, mark verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package fwupd - 1.7.9-1~20.04.1

---------------
fwupd (1.7.9-1~20.04.1) focal; urgency=medium

  * New upstream release, and drop all patches since they are merged.
  * Properly fall back to use DMI instead of /sys/class/dmi interface.
    (LP: #1982103)
  * Build depends on mondemmanager 1.8 and libxmlb 0.3.6 to support
    EM120/160. (LP: #1980334)
  * Don't stderr-fail the autopkgtest on modprobe error as they are
    optional. (LP: #1966364)
  * Run fwupd-refresh under a dedicated fwupd-refresh user. This is
    fixed in 1.1.7, so it's automatically included. (LP: #1969976)

 -- Yuan-Chen Cheng <email address hidden> Sun, 03 Jul 2022 03:18:51 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package libxmlb - 0.3.6-2build1~20.04.1

---------------
libxmlb (0.3.6-2build1~20.04.1) focal; urgency=medium

  * Backport to focal (no changes).
  * Upgrade to support EM120/160 firmware update (LP: #1980334)

 -- Yuan-Chen Cheng <email address hidden> Tue, 02 Aug 2022 07:04:04 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package gnome-software - 3.36.1-0ubuntu0.20.04.1

---------------
gnome-software (3.36.1-0ubuntu0.20.04.1) focal; urgency=medium

  * no change rebuild with libxmlb 0.3.6, which bump from libxmlb1 to
    libxmlb2. (LP: #1980334, LP: #1984840)
  * Also build-depends on fwupd 1.7.9 to make sure everything is
    consistant.

 -- Yuan-Chen Cheng <email address hidden> Mon, 08 Aug 2022 03:08:45 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package gnome-firmware - 3.34.0-2ubuntu1

---------------
gnome-firmware (3.34.0-2ubuntu1) focal; urgency=medium

  * no change rebuild with libxmlb 0.3.6, which bump from libxmlb1 to
    libxmlb2. (LP: #1980334, LP: #1984840)
  * Also build-depends on fwupd 1.7.9 to make sure everything is
    consistant.

 -- Yuan-Chen Cheng <email address hidden> Mon, 08 Aug 2022 03:13:41 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for fwupd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted fwupd (1.7.9-1~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

fwupd/1.7.9-1~20.04.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#fwupd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Utkarsh Gupta (utkarsh)]

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.