[SRU Exception] Backport 5.28 to Jammy

[Heather Ellsworth]

[Impact]

This SRU has two specific impacts: cross-architecture testing, and pocket pinning. There may be unknown impacts in backporting the version in lunar to jammy (5.20->5.28).

Initially, the patches were added to Jammy and Kinetic to provide the cross-architecture testing and pocket pinning in https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2002455 but we have since decided to go ahead and backport the version of lunar that include these patches, back to jammy. Below is a summary of the two initially sought after impacts.

1. cross-architecture testing

autopkgtest currently does not have the -a ARCH capability needed to permit cross-architecture testing. This is notably useful for i386 local testing [0] and was noted to be missing since the focal release.

The goal is to be able to create an amd64 VM where autopkgtest can run on an i386 package.

There was an existing patchset for this functionality from ~3 years ago that needed tweaking and is in a working state now[1].

2. pocket pinning (blurb copied from commit[2])

Ubuntu is switching to NotAutomatic: yes for the proposed pocket:
the change is already operative on >= Lunar and under consideration
for older stable releases. This means that by default packages from
-proposed won't be installed unless explicitly requested.

The change is meant to make it easier for end users to keep -proposed
enabled (for SRU verification or individual package installation)
without manually adding a pin to avoid pulling in all of -proposed.

However when running autopkgtests with --apt-pocket=proposed we do want
to pull in all the pocket, so let's always give it Pin-Priority: 500.
This is similar to what Launchpad does with the buildds.

Note: This change does not really treat -proposed in a special way:
any pocket will get Pin-Priority: 500. It makes sense to keep the tool
behavior consistent, and we're not aware of cases where this could be
cause issues.

[0] https://wiki.ubuntu.com/i386
[1] https://code.launchpad.net/~ubuntu-release/autopkgtest/+git/development/+merge/376169
[2] https://git.launchpad.net/~ubuntu-release/autopkgtest/+git/development/commit/?id=c92df58ae5f3bc55aa58bbfb93ca60cb38b2615c

[ Test Plan ]

1. cross-architecture testing

Create an amd64 VM where autopkgtest can run on an i386 package:

## Generate the VM
$ sudo autopkgtest-buildvm-ubuntu-cloud -a amd64 -r jammy -s 15G

## Run autopkgtest on the i386 vlc package, finishes quickly
## Note that you should run the autopkgtest binary directly (so use the path to your
## runner/autopkgtest) and point to the .img file created in the last step
$ sudo ~/autopkgtest-5.25ubuntu1/runner/autopkgtest --setup-commands="dpkg --add-architecture i386; apt-get update" --architecture i386 vlc -- qemu --ram-size=1536 --cpus 2 ~/autopkgtest-jammy-amd64.img

A passing result looks like autopkgtest setting up the testbed and launching the tests while a failing result looks something like this:

autopkgtest: error: unrecognized arguments: --architecture

2. pocket pinning testing

Using the --apt-pocket=proposed option, you should see that the proposed pocket is used when running autopkgtests.

Also if there are any packages in proposed, then you should be able to track the pocket and update without pulling in any proposed packages.

[ Where problems could occur ]

1. cross-architecture testing
It is possible for the img file to find itself in a bad state, affecting the tests run, but simply recreating the VM is a quick fix. Without this patchset i386 testing less accessible, since i386 hardware is required, so the improvement greatly outweighs potentially introduced risk.

2. pocket pinning
We are not aware of any issues that may arise from this patch.

[Heather Ellsworth (hellsworth)]

Here is a debdiff from 5.20ubuntu1 to 5.28ubuntu1.

[Heather Ellsworth (hellsworth)]

Here is a debdiff that just updates the versioning scheme to something more intuitive that also will avoid collisions with the version that exists in lunar.

[Brian Murray (brian-murray)]

I've uploaded this just now.

[Steve Langasek (vorlon)]

Hello Heather, or anyone else affected,

Accepted autopkgtest into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.28ubuntu0.22.04.0 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted autopkgtest (5.28ubuntu0.22.04.0) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/249.11-0ubuntu3.9 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Tim Andersson (andersson123)]

On my jammy machine I ran these steps to confirm the --apt-pocket=proposed option was working

lxc launch ubuntu:jammy sru-lp2008026-j
lxc exec sru-lp2008026-j bash
add "deb http://archive.ubuntu.com/ubuntu jammy-proposed main restricted" to /etc/apt/sources.list
sudo apt update
sudo apt-get -t jammy-proposed install autopkgtest
# check autopkgtest version
/home/ubuntu# apt list --installed | grep autopkgtest

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

autopkgtest/kinetic-proposed,now 5.28ubuntu0.22.10.0 all [installed]
exit
# on my machine (building image was too slow in container)
# Building mantic machine to get proposed version of isc-dhcp package
sudo autopkgtest-buildvm-ubuntu-cloud -a amd64 -r mantic -s 15G
lxc file push autopkgtest-mantic-amd64.img sru-lp2008026-j/
# back to container
lxc exec sru-lp2008026-j bash
sudo autopkgtest --setup-commands="apt-get update" --apt-pocket=proposed isc-dhcp -- qemu --ram-size=1536 --cpus 2 ./autopkgtest-mantic-amd64.img

Relevant output from autopkgtest:
Get:1 http://archive.ubuntu.com/ubuntu mantic-proposed/main isc-dhcp 4.4.3-P1-1ubuntu2 (dsc) [2,969 B]
Get:2 http://archive.ubuntu.com/ubuntu mantic-proposed/main isc-dhcp 4.4.3-P1-1ubuntu2 (tar) [10.1 MB]
Get:3 http://archive.ubuntu.com/ubuntu mantic-proposed/main isc-dhcp 4.4.3-P1-1ubuntu2 (asc) [874 B]
Get:4 http://archive.ubuntu.com/ubuntu mantic-proposed/main isc-dhcp 4.4.3-P1-1ubuntu2 (diff) [129 kB]
gpgv: Signature made Thu 11 May 2023 03:13:31 PM BST
gpgv: using RSA key D98F377EB47CC6DD257A9A07E564B9C275BDD52E
gpgv: Can't check signature: No public key
dpkg-source: warning: cannot verify inline signature for ./isc-dhcp_4.4.3-P1-1ubuntu2.dsc: no acceptable signature found
autopkgtest [12:18:02]: testing package isc-dhcp version 4.4.3-P1-1ubuntu2

rmadison output to match version:
rmadison isc-dhcp
 isc-dhcp | 4.2.4-7ubuntu12 | trusty | source
 isc-dhcp | 4.2.4-7ubuntu12.12 | trusty-security | source
 isc-dhcp | 4.2.4-7ubuntu12.13 | trusty-updates | source
 isc-dhcp | 4.3.3-5ubuntu12 | xenial | source
 isc-dhcp | 4.3.3-5ubuntu12.9 | xenial-security | source
 isc-dhcp | 4.3.3-5ubuntu12.10 | xenial-updates | source
 isc-dhcp | 4.3.5-3ubuntu7 | bionic | source
 isc-dhcp | 4.3.5-3ubuntu7.4 | bionic-security | source
 isc-dhcp | 4.3.5-3ubuntu7.4 | bionic-updates | source
 isc-dhcp | 4.4.1-2.1ubuntu5 | focal | source
 isc-dhcp | 4.4.1-2.1ubuntu5.20.04.4 | focal-security | source
 isc-dhcp | 4.4.1-2.1ubuntu5.20.04.5 | focal-updates | source
 isc-dhcp | 4.4.1-2.3ubuntu2 | jammy | source
 isc-dhcp | 4.4.1-2.3ubuntu2.3 | jammy-security | source
 isc-dhcp | 4.4.1-2.3ubuntu2.4 | jammy-updates | source
 isc-dhcp | 4.4.3-2ubuntu4 | kinetic | source
 isc-dhcp | 4.4.3-P1-1ubuntu1 | lunar | source
 isc-dhcp | 4.4.3-P1-1ubuntu1 | mantic | source
 isc-dhcp | 4.4.3-P1-1ubuntu2 | mantic-proposed | source

We can see that autopkgtest is fetching version 4.4.3-P1-1ubuntu2 of isc-dhcp from mantic-proposed ...

[Tim Andersson (andersson123)]

For testing the --architecture I did this:

lxc launch ubuntu:jammy sru-lp2008026-j
lxc exec sru-lp2008026-j bash
add "deb http://archive.ubuntu.com/ubuntu jammy-proposed main restricted" to /etc/apt/sources.list
sudo apt update
sudo apt-get -t jammy-proposed install autopkgtest
exit
# on my machine (building image was too slow in container)
# Building mantic machine to get proposed version of isc-dhcp package
sudo autopkgtest-buildvm-ubuntu-cloud -a amd64 -r jammy -s 15G
lxc file push autopkgtest-jammy-amd64.img sru-lp2008026-j/
# back to container
lxc exec sru-lp2008026-j bash
sudo autopkgtest --setup-commands="dpkg --add-architecture i386; apt-get update" --architecture i386 vlc -- qemu --ram-size=1536 --cpus 2 ./autopkgtest-jammy-amd64.img

However, unlike in the autopkgtest-cloud environment:
https://autopkgtest.ubuntu.com/packages/vlc
vlc-all test fails for me on the vlc-all test
badpkg: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.
autopkgtest [12:43:19]: @@@@@@@@@@@@@@@@@@@@ summary
vlc-bin PASS
vlc PASS
vlc-all FAIL badpkg
blame: vlc
badpkg: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.

Despite having made these .img files today, so I'm not sure where I'm going wrong

[Tim Andersson (andersson123)]

Just as an addendum comment, I tested the --add-architecture arg in a kinetic container and vlc-all did not fail, so I think there may be an issue

[Tim Andersson (andersson123)]

Testing on jammy with version 5.25:

root@timscontainer:/home/ubuntu# apt list --installed | grep autopkgtest

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

autopkgtest/kinetic,now 5.25ubuntu1 all [installed]
root@timscontainer:/home/ubuntu# sudo autopkgtest --architecture i386 vlc -- qemu --ram-size=1536 --cpus 2 ./autopkgtest-jammy-amd64.img
usage: autopkgtest [options] [testbinary ...] testsrc -- virt-server [options]
autopkgtest: error: unrecognized arguments: --architecture

[Tim Andersson (andersson123)]

Retesting the --architecture arg with mawk as vlc has issues with the autopkgtest tests failing, plus mawk is a lot faster

lxc launch ubuntu:jammy timscontainer
lxc exec timscontainer bash
sudo apt install autopkgtest
root@timscontainer:/home/ubuntu# apt list --installed | grep autopkgtest

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

autopkgtest/jammy-updates,now 5.20ubuntu1 all [installed]
root@timscontainer:/home/ubuntu# sudo autopkgtest --setup-commands="dpkg --add-architecture i386; apt-get update" --architecture i386 mawk -- qemu --ram-size=1536 --cpus 2 ./autopkgtest-jammy-amd64.img
usage: autopkgtest [options] [testbinary ...] testsrc -- virt-server [options]
autopkgtest: error: unrecognized arguments: --architecture

# install correct version
root@timscontainer:/home/ubuntu# apt list --installed | grep autopkgtest

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

autopkgtest/jammy-proposed,now 5.28ubuntu0.22.04.0 all [installed]

sudo autopkgtest --setup-commands="dpkg --add-architecture i386; apt-get update" --architecture i386 mawk -- qemu --ram-size=1536 --cpus 2 ./autopkgtest-jammy-amd64.img

Output:
testing function calls and general stress test
general stress test OK
tested mawk OK
autopkgtest [15:48:09]: test mawktest: -----------------------]
autopkgtest [15:48:10]: test mawktest: - - - - - - - - - - results - - - - - - - - - -
mawktest PASS
autopkgtest [15:48:11]: @@@@@@@@@@@@@@@@@@@@ summary
mawktest PASS
qemu-system-x86_64: terminating on signal 15 from pid 4345 (/usr/bin/python3)

And it all worked perfectly fine! The vlc package was the problem in the previous comment.

[Robie Basak (racb)]

What's the state of the fix in Mantic, please?

[Tim Andersson (andersson123)]

(testing on Mantic)

[Brian Murray (brian-murray)]

On Wed, May 24, 2023 at 10:32:01AM -0000, Robie Basak wrote:
> What's the state of the fix in Mantic, please?

5.28 is available in Mantic so it is Fix Released.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2008026
>
> Title:
> [SRU Exception] Backport 5.28 to Jammy
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2008026/+subscriptions
>
--
Brian Murray

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 5.28ubuntu0.22.04.0

---------------
autopkgtest (5.28ubuntu0.22.04.0) jammy; urgency=medium

  [ Paride Legovini ]
  * Backport version 5.28 of autopkgtest. (LP: #2008026)
  * Include patches for cross-arch testing and preventing --apt-pocket from
    setting the wrong default release

 -- Heather Ellsworth <email address hidden> Tue, 14 Mar 2023 16:04:39 -0600

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for autopkgtest has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.