Support manual firmware upgrading for Foxconn and Quectel modems.

[Jerry Lee]

[Impact]

The modem certification requires that different modem firmware is used for different network carrier.
This needs the firmware upgrading capability during the modem certification process.

The modem manufacture vendors (Foxconn and Quectel) provided utilities to do modem's firmware upgrading manually.(LP#1943774, LP#1943780)
These utilities are verified to be working when the recent versions(> v 1.18.2) of ModemManager are used with.

To support manual firmware upgrading on the current Focal release which is using ModemManager v 1.16.6, we need to apply some patches from v 1.18.2.
The requested upstream patches are listed as below:
* for Quectel EM160 4G
  ** https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/83ac82470589a3672092a0ba0be855093b1cf5e2
* for Foxconn T99W175
  ** https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/21ae558fe3600c84b3ca7dcd9bf50a3ba576c7c9
  **https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/76e700f4fd703f952208993330ab098305c13d6b
  ** https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/52bf2c641171ded9e617022f40497c8984520371
  ** https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/33e2b023ef01bea9da37ae2beb192f7d92bce47a
  ** https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/commit/f72046701659073fbfa97516e155865647acb154

The firmware upgrading was verified using the patched ModemManager v 1.16.6 with the following 2 modems:
* Foxconn SDX55 T99W175 5G sub6 PCIE Modem
* Quectel SDX24 EM160R-GL 4G LTE CAT16 PCIE Modem

[Test Plan]

1. Install the Ubuntu image.
2. Boot and login the system.
3. Verify if the modem is working
4. Prepare the modem’s firmware and install the firmware upgrading application provided by Foxconn and Quectel
5. Using the firmware upgrading application to upgrade the modem’s firmware
6. Verify if the modem’s firmware upgrading is successful
7. Reboot
8. Verify if the upgraded modem firmware is still working

--- Regression test ---

Verify if one USB modem are still working with these patches for PCIe modems.

[Where problems could occur]

The requested update has 2 parts:

1. Informative
   1.1 Provide more information about modems whose drivers use WWAN subsystem in kernel 5.13
   1.2 Modem manufacture's private utilities can use this information to do modem's FW upgrading manually

2. Changes are specific to Foxconn and Quectel modems
   2.1 Modified code are only used by Foxconn and Quectel modems during their FW upgrading. (matched by vendor_id and product_id)

In current Ubuntu's certification records for modem:
* No other modem uses WWAN subsystem in kernel 5.13
* Modem's FW update is not supported via ModemManager ( < v1.18 )

There is no certificated modems can do the firmware upgrading flow for the regression test.
This update should not affect existing modems.

The problem would be limited to these two mentioned modems.
Each carrier mapping .conf file is for a specific modem.
ModemManager will load one of the carrier mapping conf files via the modem manufacturer’s plugin ( if the PCIe VID & PID is matched by the plugin.)
We cannot verify if the carrier mapping is correct. This relies on the manufacturer to provide the correct mapping.

The carrier mapping .conf files is verified by modem’s manufacture according to the tested SIM card published by different countries.
Modem manufacturer confirmed that the content in the .conf file is absolutely correct.

[Other Info]

The firmware and the upgrading utilities can be downloaded from the following link:
* LP#1943774 for Quectel modems
* LP#1943780 for Foxconn modems

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-3-focal.debdiff” is attached for focal(20.04).

The pre-built package : https://launchpad.net/~jerry-lee-tpe/+archive/ubuntu/packages/+packages

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-3-hirsute.debdiff” is attached for hirsute(21.04).

The pre-built package :  https://launchpad.net/~jerry-lee-tpe/+archive/ubuntu/packages/+packages

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "modemmanager_1-1.16.6-3-focal.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Sebastien Bacher (seb128)]

would you have an update for impish, that's where we should start

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-3-impish.debdiff” is attached for impish(22.04).

The pre-built package : https://launchpad.net/~jerry-lee-tpe/+archive/ubuntu/packages/+packages

[Sebastien Bacher (seb128)]

Upload to impish, hirsute and focal. Please try to remember including the bug reference in the changelog next time though, I've fixed it before uploading for this round

[Jerry Lee (jerry-lee-tpe)]

Hi Sebastien,

I verified this new package from impish-proposed :

* The version of the packages tested:
   ModemManager: 1.16.6-2‌ubuntu1

* The test report is attached as "TestReport_impish-proposed_1946096.pdf".

Besides, I ave not found the packages are ready in hirsute-proposed and focal-proposed yet.
Can you help to check it?

Thanks

[Sebastien Bacher (seb128)]

The stable queue needs review from the SRU team

https://launchpad.net/ubuntu/hirsute/+queue?queue_state=1

for example

[Jerry Lee (jerry-lee-tpe)]

Hi SRU team,

Please help to check the status of this case which looks stuck at some steps.

Thanks

[Robie Basak (racb)]

SRU review

I don't see any explanation of what this is fixing, or why this is needed, so cannot review if the proposed changes qualify for an SRU under Ubuntu's policies.

Specifically I need an explanation of actual user impact, not just "fix some problems". Further, note that the procedure says "Do not create a meta-bug with a title like "Please SRU this" rather than using existing bug reports". We need actual public bugs for each of the specific problems that you're fixing. Each of those bugs should be linked from the changelog, so we can ensure during SRU verification that each of those issues have individually been verified fixed. Done properly, there is then no need for a meta bug like this one.

See https://wiki.ubuntu.com/StableReleaseUpdates#Procedure for full details on what is expected.

The Focal upload is also missing a proper bug reference - the reference given doesn't match the standard pattern so didn't get copied to the Launchpad-Bugs-Fixed header.

I'm rejecting both the Hirsute and Focal uploads since both uploads need their changelogs fixing to properly reference public bugs that contain full SRU information.

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-2~21.04.1-hirsute.debdiff” is attached for hirsute(21.04).

The pre-built package : https://launchpad.net/~jerry-lee-tpe/+archive/ubuntu/packages/+packages

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-2~20.04.1-focal.debdiff” is attached for focal(20.04).

The pre-built package : https://launchpad.net/~jerry-lee-tpe/+archive/ubuntu/packages/+packages

[Łukasz Zemczak (sil2100)]

Sponsored both the focal and hirsute versions.

[Steve Langasek (vorlon)]

Is there a risk that applying these changes will regress the behavior of these modems on systems which have NOT applied the firmware updates? Can the testcase include a check for this?

[Jerry Lee (jerry-lee-tpe)]

Applying these changes should have no risk:
* These changes are only for specific modems.(Foxconn and Quecktel)
* These changes are used when the platform's vendors do the firmware upgrading for specific modems "manually".

We can try to find other modems then provide the regression test report later.

[Jerry Lee (jerry-lee-tpe)]

These patches are used for newer Foxconn/Quectel modems :
* whose drivers are using the WWAN subsystem in the kernel 5.13. ( this OEM project uses customized kernel )
* communicated to the ModemManager service via the MBIM protocol

These patches will not be used if the modem can't meet the above requirements.

After checking Ubuntu certificated modems, there is no modem can meet these 2 requirement to do the regression test.

[Chris Halse Rogers (raof)]

Ok. For clarity, my understanding of these patches are that they do the following things:
1) Add a feature to ModemManager to support uploading firmware to these modems
2) Increase a timeout that was found to be insufficient when doing a firmware update
3) Add a bunch of mappings from SIM->Carrier

And then in response to potential concerns:
(1) is a straight feature addition, and the associated code will only be called if an external component connects to modemmanager requesting a firmware upload. This is not expected to change any existing behaviours.
(2) is low-risk

It's not clear to me whether (3) is expected to change existing behaviours at all.

Is this understanding correct?

Assuming (3) is not expected to change existing behaviour this looks like it would be acceptable under the hardware-enablement SRU policy.

[Jerry Lee (jerry-lee-tpe)]

Yes, (3) is not expected to change existing behavior.

[Robie Basak (racb)]

> 3) Add a bunch of mappings from SIM->Carrier

> Assuming (3) is not expected to change existing behaviour this looks like it would be acceptable under the hardware-enablement SRU policy.

Agreed, but I would expect SRU information against every change being made, that would then receive separate SRU verification. This helps us mitigate risk by 1) actually considering each change in the first place, rather than it "slipping past" review; and 2) reducing the potential number of SRUs, and therefore user time, bandwidth and regression risk, by actually verifying that things are being changed as expected so we don't have to do it twice.

I'm disappointed that this has now gone through two SRU review rounds, and it is Chris who has extracted the actual nature of the changes that you are looking to land here. This detail should have been laid out in the bug (or multiple bugs) and in the SRU information in the first place.

[Robie Basak (racb)]

How do you intend to verify that (3) has worked correctly?

[Jerry Lee (jerry-lee-tpe)]

By investigating the mentioned patches directly, the existing behavior should not be changed:

* The patch for Quectel EM160 4G
  There is only one patch, all changes are for the Quectel modems.

* Patches for Foxconn T99W175
  There are 5 patches,
  (patch#1): Modify the command line tool to get additional scan report for WWAN subsystem
  (patch#2): Move methods between 2 files
  (patch#3): Add more 15 seconds wait for the firmware upgrading
  (patch#4): Rename an existed carrier mapping configuration file for Foxconn modem
  (patch#5): Add a new carrier mapping configuration file for Foxconn modem

From the code review, it's clear that changes only affect Foxconn and Quectel modems.

 As "3) Add a bunch of mappings from SIM->Carrier":

(a) The 2 modified carrier mapping config files are for Foxconn modems
(b) These 2 config files are used in the changed source file by hard coded
(c) Only matched(vendor_id, product_id) Foxconn's modem will use these 2 config files

[Robie Basak (racb)]

OK, so you currently have nothing in your Test Plan to verify this part of the changes you're proposing? Please either fix the Test Plan, or if testing this is impractical for some reason then we can discuss that and decide what mitigation might be appropriate, but in this case the omission and conclusion should be documented in this bug - presumably in the "Where problems could occur" section.

[Jerry Lee (jerry-lee-tpe)]

[Where problems could occur] is updated

[Robie Basak (racb)]

I don't see how your updated "Where problems could occur" addresses my comment 22 at all.

[Jerry Lee (jerry-lee-tpe)]

[Where problems could occur] is updated

[Jerry Lee (jerry-lee-tpe)]

Hello, is there anything needs me to provide for moving to the next step?

[Robie Basak (racb)]

Before I was relying on the information presented here and the changelog in the upload to consider regression risk. Today I reviewed the patches in more detail expecting to find what has been described. What's going on in 01_modem-quectel.patch? This says "add support for EM120/160 PCIe modules" and it seems to be adding support for a PCIe variant of a modem in the code that already supports a USB variant. But your changelog just says "Support manual firmware upgrading" with no mention of support for additional modems. Are you actually additionally adding support for this modem hardware where support didn't exist previously? I see no mention of additional additional modem support anywhere in this bug so far.

If so, then will you be able to test the regular functionality of these modems - both USB and PCIe - during SRU verification please? Do you have these two modems available? This would be to ensure that 1) the changes to modemmanager that relate to the USB variant haven't regressed; and 2) the changes to add support for the PCIe variant is actually working. If you can do this, then please add this to the Test Plan. If not, then further discussion is needed.

It looks like Impish no longer carries this patch as 1.16.6-2ubuntu1 was removed, so we also need an upload of this to Impish please.

[Jerry Lee (jerry-lee-tpe)]

The "01_modem-quectel.patch" extends the quectel:plugin to provide firmware setting status during the FW upgrading process.
EM120/160 PCIe modules have been already supported without this patch.
I got the confirmation from Quectel engineers about the above.

For the regression test, we found 2 laptops having Quectel USB modems and did the test on these laptops.
The Quectel USB modems worked with the patched ModemManager.
The regression test report is attached here: "RegressionTestReport_ EM05-CE.pdf".

[Jerry Lee (jerry-lee-tpe)]

The .debdiff file “modemmanager_1-1.16.6-2~21.10.1-impish.debdiff” is attached for impish(21.10).

[Robie Basak (racb)]

I think all concerns raised have now been addressed so I'm accepting the upload into Hirsute and Focal.

[Robie Basak (racb)]

Hello Jerry, or anyone else affected,

Accepted modemmanager into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/modemmanager/1.16.6-2~21.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Jerry, or anyone else affected,

Accepted modemmanager into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/modemmanager/1.16.6-2~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Fixed in Jammy in 1.18.2-0ubuntu1.

[Robie Basak (racb)]

(and in Impish in 1.16.6-2ubuntu1)

[Jerry Lee (jerry-lee-tpe)]

(1) Test passed with Lenovo ThinkEdge SE30(11ND) and Lenovo ThinkEdge SE30(11NE) on Hirsute:
   * The version of the packages tested:
     ModemManager: 1.16.6-2~21.04.1

(2) Test passed with Lenovo ThinkEdge SE30(11ND) and Lenovo ThinkEdge SE30(11NE) on Focal:
   * The version of the packages tested:
     ModemManager: 1.16.6-2~20.04.1

The detail testing procedure and result is provided as the attached file : "TestReportForHirsuteAndFocal-proposed-1946096.pdf"

The regression test report will be provided.

[Jerry Lee (jerry-lee-tpe)]

Regression test results:

(1) Test passed with Lenovo T14s (Tiger-2) on Hirsute:
   * The version of the packages tested:
     ModemManager: 1.16.6-2~21.04.1

(2) Test passed with Lenovo X13 AMD (Servel-AMD-1) on Focal:
   * The version of the packages tested:
     ModemManager: 1.16.6-2~20.04.1

The detail testing procedure and result is provided as the attached file : "RegressionTestReportFor-proposedPackages-1946096.pdf"

[Launchpad Janitor (janitor)]

This bug was fixed in the package modemmanager - 1.16.6-2~21.04.1

---------------
modemmanager (1.16.6-2~21.04.1) hirsute; urgency=medium

  * Support manual firmware upgrading for Foxconn and Quectel modems.
    (LP: #1946096)

 -- Jerry Lee <email address hidden> Thu, 21 Oct 2021 10:43:42 +0800

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for modemmanager has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package modemmanager - 1.16.6-2~20.04.1

---------------
modemmanager (1.16.6-2~20.04.1) focal; urgency=medium

  * Support manual firmware upgrading for Foxconn and Quectel modems.
    (LP: #1946096)

 -- Jerry Lee <email address hidden> Thu, 21 Oct 2021 10:29:18 +0800