Fix sanity checker and cleanup metadata

[Juerg Haefliger]

The repo contains a checker script check_whence.py that checks WHENCE data against present firmware files and flags missing files and missing WHENCE entries. It currently doesn't parse the downstream WHENCE.ubuntu file. Fix that and cleanup any errors flagged by the checker.

[ Impact ]

None. These are packaging changes that are not visible to the end user.

[ Where Problems Could Occur ]

None.

[Timo Aaltonen (tjaalton)]

Hello Juerg, or anyone else affected,

Accepted linux-firmware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.187.23 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Juerg, or anyone else affected,

Accepted linux-firmware into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.197.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Juerg, or anyone else affected,

Accepted linux-firmware into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.201.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Juerg Haefliger (juergh)]

Packages contain updated WHENCE files to verification done on F/H/I.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for linux-firmware has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.197.5

---------------
linux-firmware (1.197.5) hirsute; urgency=medium

  * linux-firmware: add firmware for adl machines from sof-firmware-v1.9.2 (LP: #1952011)
  * Miscellaneous Ubuntu packaging changes and cleanups (LP: #1953587)
    - SAUCE: check_whence.py: Add python3 shebang
    - SAUCE: check_whence.py: Also process WHENCE.ubuntu
    - SAUCE: check_whence.py: Ignore debian/ and fw_source/ directories
    - SAUCE: check_whence.py: Ignore ea/ directory
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete tr_smctr from WHENCE
    - SAUCE: Remove obsolete ti-connectivity from WHENCE
    - SAUCE: Remove obsolete tda7706 from WHENCE
    - SAUCE: Remove obsolete sxg from WHENCE
    - SAUCE: Remove obsolete myricom from WHENCE
    - SAUCE: Remove obsolete mwl8k from WHENCE
    - SAUCE: Remove obsolete libertas from WHENCE
    - SAUCE: Remove obsolete Computone from WHENCE
    - SAUCE: Remove obsolete dabusb from WHENCE
    - SAUCE: Remove obsolete cxgb from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete ar9170 from WHENCE
    - SAUCE: Remove obsolete ar7010 from WHENCE
    - SAUCE: Remove obsolete ath3k from WHENCE
    - SAUCE: Remove obsolete 3com from WHENCE
    - SAUCE: Remove obsolete ath6k from WHENCE
    - SAUCE: Remove obsolete slicoss from WHENCE
    - SAUCE: Remove obsolete ath9k from WHENCE
    - SAUCE: Remove obsolete ar3k from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete files from WHENCE
    - SAUCE: Remove obsolete cxgb3 from WHENCE
    - SAUCE: Remove obsolete ueagle-atm from WHENCE
    - Add LICENCE.realtek-firmware.txt to WHENCE.ubuntu
    - Rename LICENCE.dib0700 to LICENSE.dib0700
    - Move Ubuntu check_firmware script to debian/scripts/
    - Remove unnecessary symbolic links
    - intel/sof: Remove symbolic links and add license to WHENCE.ubuntu
    - [Packaging] Add rebase-upstream helper script and debian rule
    - [Packaging] Add close-release helper script and debian rule
    - [Packaging] Add check rule and call it in clean stage
    - SAUCE: Remove missing firmwares from WHENCE
    - [Packaging] Add WHENCE.ubuntu as README.ubuntu to the package
    - [Packaging] close-release: Parse commit messages for Ignores and BugLinks

 -- Juerg Haefliger <email address hidden> Tue, 07 Dec 2021 13:26:28 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.201.3

---------------
linux-firmware (1.201.3) impish; urgency=medium

  * linux-firmware: add firmware for adl machines from sof-firmware-v1.9.2 (LP: #1952011)
  * Miscellaneous Ubuntu packaging changes and cleanups (LP: #1953587)
    - SAUCE: check_whence.py: Add python3 shebang
    - SAUCE: check_whence.py: Also process WHENCE.ubuntu
    - SAUCE: check_whence.py: Ignore debian/ and fw_source/ directories
    - SAUCE: check_whence.py: Ignore ea/ directory
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete tr_smctr from WHENCE
    - SAUCE: Remove obsolete ti-connectivity from WHENCE
    - SAUCE: Remove obsolete tda7706 from WHENCE
    - SAUCE: Remove obsolete sxg from WHENCE
    - SAUCE: Remove obsolete myricom from WHENCE
    - SAUCE: Remove obsolete mwl8k from WHENCE
    - SAUCE: Remove obsolete libertas from WHENCE
    - SAUCE: Remove obsolete Computone from WHENCE
    - SAUCE: Remove obsolete dabusb from WHENCE
    - SAUCE: Remove obsolete cxgb from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete ar9170 from WHENCE
    - SAUCE: Remove obsolete ar7010 from WHENCE
    - SAUCE: Remove obsolete ath3k from WHENCE
    - SAUCE: Remove obsolete 3com from WHENCE
    - SAUCE: Remove obsolete ath6k from WHENCE
    - SAUCE: Remove obsolete slicoss from WHENCE
    - SAUCE: Remove obsolete ath9k from WHENCE
    - SAUCE: Remove obsolete ar3k from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete files from WHENCE
    - SAUCE: Remove obsolete cxgb3 from WHENCE
    - SAUCE: Remove obsolete ueagle-atm from WHENCE
    - Add LICENCE.realtek-firmware.txt to WHENCE.ubuntu
    - Rename LICENCE.dib0700 to LICENSE.dib0700
    - Move Ubuntu check_firmware script to debian/scripts/
    - Remove unnecessary symbolic links
    - intel/sof: Remove symbolic links and add license to WHENCE.ubuntu
    - [Packaging] Add check rule and call it in clean stage
    - [Packaging] Add WHENCE.ubuntu as README.ubuntu to the package
    - UBUNTU: [Packaging] close-release: Parse commit messages for Ignores and BugLinks

 -- Juerg Haefliger <email address hidden> Tue, 07 Dec 2021 09:40:38 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.187.23

---------------
linux-firmware (1.187.23) focal; urgency=medium

  * i915: Add HuC firwmare v7.5.0 for TGL (LP: #1939986)
  * i915: Add DG1 DMC v2.02 (LP: #1939986)
  * i915: Add GuC firmware v49.0.1 for all platforms (LP: #1939986)
  * amdgpu: update yellow carp dmcub firmware (LP: #1953008)
  * linux-firmware: add firmware for adl machines from sof-firmware-v1.9.2 (LP: #1952011)
  * Miscellaneous Ubuntu packaging changes and cleanups (LP: #1953587)
    - SAUCE: check_whence.py: Add python3 shebang
    - SAUCE: check_whence.py: Also process WHENCE.ubuntu
    - SAUCE: check_whence.py: Ignore debian/ and fw_source/ directories
    - SAUCE: check_whence.py: Ignore ea/ directory
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete tr_smctr from WHENCE
    - SAUCE: Remove obsolete ti-connectivity from WHENCE
    - SAUCE: Remove obsolete tda7706 from WHENCE
    - SAUCE: Remove obsolete sxg from WHENCE
    - SAUCE: Remove obsolete myricom from WHENCE
    - SAUCE: Remove obsolete mwl8k from WHENCE
    - SAUCE: Remove obsolete libertas from WHENCE
    - SAUCE: Remove obsolete Computone from WHENCE
    - SAUCE: Remove obsolete dabusb from WHENCE
    - SAUCE: Remove obsolete cxgb from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete ar9170 from WHENCE
    - SAUCE: Remove obsolete ar7010 from WHENCE
    - SAUCE: Remove obsolete ath3k from WHENCE
    - SAUCE: Remove obsolete 3com from WHENCE
    - SAUCE: Remove obsolete ath6k from WHENCE
    - SAUCE: Remove obsolete slicoss from WHENCE
    - SAUCE: Remove obsolete ath9k from WHENCE
    - SAUCE: Remove obsolete ar3k from WHENCE
    - SAUCE: Remove obsolete bnx2x from WHENCE
    - SAUCE: Remove obsolete files from WHENCE
    - SAUCE: Remove obsolete cxgb3 from WHENCE
    - SAUCE: Remove obsolete ueagle-atm from WHENCE
    - Add LICENCE.realtek-firmware.txt to WHENCE.ubuntu
    - Rename LICENCE.dib0700 to LICENSE.dib0700
    - Move Ubuntu check_firmware script to debian/scripts/
    - Remove unnecessary symbolic links
    - intel/sof: Remove symbolic links and add license to WHENCE.ubuntu
    - SAUCE: Remove missing firmwares from WHENCE
    - Fix ipu6 firmware license reference
    - [Packaging] Add check rule and call it in clean stage
    - [Packaging] close-release: Parse commit messages for Ignores and BugLinks

 -- Juerg Haefliger <email address hidden> Wed, 08 Dec 2021 08:11:55 +0100

[Timo Aaltonen (tjaalton)]

Hello Juerg, or anyone else affected,

Accepted linux-firmware into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware/1.173.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mauricio Faria de Oliveira (mfo)]

Verification done for bionic-proposed.

There are no verification steps, but the description shows what to check for:
- It currently doesn't parse the downstream WHENCE.ubuntu file.
- Fix that and cleanup any errors flagged by the checker.

Note: these must be performed on the git repo per dep on `git ls-files`.

PS: Juergh, hope you don't mind someone else verifying this!
Otherwise, sorry, please feel free to revert the tag change.

bionic-updates:
---

$ git checkout Ubuntu-1.173.20

WHENCE.ubuntu? No.

$ strace -f -e openat -o open.log ./check_whence.py 2>/dev/null
$ grep WHENCE open.log
2725 openat(AT_FDCWD, "WHENCE", O_RDONLY) = 3

Clean of errors? No.

$ ./check_whence.py 2>&1 | awk '{ $2 = "FILE"; print $0 }' | sort | uniq -c
    131 E: FILE listed in WHENCE does not exist
    115 E: FILE not listed in WHENCE

bionic-proposed:
---

$ git checkout Ubuntu-1.173.21

WHENCE.ubuntu? Yes.

$ strace -f -e openat -o open.log ./check_whence.py 2>/dev/null
$ grep WHENCE open.log
2755 openat(AT_FDCWD, "WHENCE", O_RDONLY|O_CLOEXEC) = 3
2755 openat(AT_FDCWD, "WHENCE.ubuntu", O_RDONLY|O_CLOEXEC) = 3

Clean of errors? Yes.

$ ./check_whence.py 2>&1 | awk '{ $2 = "FILE"; print $0 }' | sort | uniq -c
$

What if we force one? It works.

$ vim WHENCE # remove last entry
$ git diff
diff --git a/WHENCE b/WHENCE
...
-
-Driver: ice - Intel(R) Ethernet Connection E800 Series
-
-File: intel/ice/ddp/ice-1.3.4.0.pkg
-Link: intel/ice/ddp/ice.pkg -> intel/ice/ddp/ice-1.3.4.0.pkg
-
-License: Redistributable. See LICENSE.ice for details

$ ./check_whence.py
E: LICENSE.ice not listed in WHENCE or WHENCE.ubuntu
E: intel/ice/ddp/ice-1.3.4.0.pkg not listed in WHENCE or WHENCE.ubuntu
E: intel/ice/ddp/ice.pkg not listed in WHENCE or WHENCE.ubuntu

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware - 1.173.21

---------------
linux-firmware (1.173.21) bionic; urgency=medium

  * Fix sanity checker and cleanup metadata (LP: #1953587)
    - Makefile: improve `make check` usefulness
    - check_whence: python3/utf-8 support
    - SAUCE: check_whence.py: Add python3 shebang
    - SAUCE: check_whence.py: Also process WHENCE.ubuntu
    - SAUCE: check_whence.py: Ignore debian/ and fw_source/ directories
    - SAUCE: check_whence.py: Ignore ea/ directory
    - Fix File: entries in WHENCE.ubuntu
    - SAUCE: Remove non-existing files from WHENCE
    - Remove non-existing files from WHENCE.ubuntu
    - SAUCE: Add missing files entries to WHENCE
    - Add missing entries to WHENCE.ubuntu
    - Remove unnecessary file LICENCE.tda7706-firmware.txt
  * Miscellaneous Ubuntu changes
    - [Packaging] Move scripts to debian/scripts
    - [Packaging] Add check rule and call it in clean stage
    - [Packaging] Add packaging helper scripts
  * Missing firmware /lib/firmware/intel/ice/ddp/ice.pkg for ice driver (LP: #1960839)
    - ice: Add package file for Intel E800 series driver

 -- Juerg Haefliger <email address hidden> Fri, 18 Feb 2022 15:59:21 +0100