autopkgtests fail due to new update-motd snippet

[Brian Murray]

ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/88-esm-announce: /etc/update-motd.d/88-esm-announce
ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/91-contract-ua-esm-status: /etc/update-motd.d/91-contract-ua-esm-status
autopkgtest [09:35:08]: test update-motd: -----------------------]
autopkgtest [09:35:08]: test update-motd: - - - - - - - - - - results - - - - - - - - - -
update-motd FAIL stderr: ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/88-esm-announce: /etc/update-motd.d/88-esm-announce
autopkgtest [09:35:08]: test update-motd: - - - - - - - - - - stderr - - - - - - - - - -
ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/88-esm-announce: /etc/update-motd.d/88-esm-announce
ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/91-contract-ua-esm-status: /etc/update-motd.d/91-contract-ua-esm-status

This output is produced by debian/tests/update-motd.

[Utkarsh Gupta (utkarsh)]

Hm, simply running autopkgtest for update-motd fails for me, too:

```
autopkgtest [13:20:24]: test show-motd: [-----------------------
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Welcome to Ubuntu 21.04 (GNU/Linux 5.8.0-50-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

                         .--._ _.--.
                        ( \ / )
                         \ /\ /
                          \_ \/ _/
                           / \
                          ( /\ )
                           `--' `--'

                           FreedomBox

FreedomBox is a pure blend of Debian GNU/Linux. Web interface is available at
https://localhost/ . FreedomBox manual is available in /usr/share/doc/freedombox
and from the web interface.

Append your ssh key to your server's authorized keys file using the command
'ssh-copy-id user@server_address'

The list of available updates is more than a week old.
To check for new updates run: sudo apt update

*** System restart required ***

This message is shown once a day. To disable it please create the
/home/ubuntu/.hushlogin file.
autopkgtest [13:20:24]: test show-motd: -----------------------]
autopkgtest [13:20:25]: test show-motd: - - - - - - - - - - results - - - - - - - - - -
show-motd PASS
autopkgtest [13:20:25]: test update-motd: preparing testbed
Get:1 file:/tmp/autopkgtest.tDxwwb/binaries InRelease
Ign:1 file:/tmp/autopkgtest.tDxwwb/binaries InRelease
Get:2 file:/tmp/autopkgtest.tDxwwb/binaries Release [816 B]
Get:2 file:/tmp/autopkgtest.tDxwwb/binaries Release [816 B]
Get:3 file:/tmp/autopkgtest.tDxwwb/binaries Release.gpg
Ign:3 file:/tmp/autopkgtest.tDxwwb/binaries Release.gpg
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/7,684 B of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 34421 files and directories currently installed.)
Preparing to unpack .../update-motd_3.7_all.deb ...
Unpacking update-motd (3.7) over (3.7) ...
Preparing to unpack .../archives/show-motd_3.7_all.deb ...
Unpacking show-motd (3.7) over (3.7) ...
Setting up update-motd (3.7) ...
Setting up show-motd (3.7) ...

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
Reading package lists...
Building dependency tree...
Reading state information...
Starting pkgProblemResolver with broken count: 0
Starting 2 pkgProblemResolver with broken count: 0
Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up autopkgtest-satdep (0) ...

No servi...

[Utkarsh Gupta (utkarsh)]

Oh wait, using qemu it passes:

autopkgtest [13:31:49]: test update-motd: -----------------------]
autopkgtest [13:31:50]: test update-motd: - - - - - - - - - - results - - - - - - - - - -
update-motd PASS
autopkgtest [13:31:51]: @@@@@@@@@@@@@@@@@@@@ summary
show-motd PASS
update-motd PASS

..but not in a container. Is it simply about adding the right "Restrictions"?

[Christian Ehrhardt  (paelzer)]

Interesting but that is a fail in a VM
https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/amd64/u/update-motd/20210430_205617_c8474@/log.gz

And https://autopkgtest.ubuntu.com/packages/update-motd looks broken on all architectures.

[Utkarsh Gupta (utkarsh)]

The above was in a Hirsute image. However it passes for Impish image, too:

```
autopkgtest [13:50:20]: test update-motd: -----------------------]
autopkgtest [13:50:21]: test update-motd: - - - - - - - - - - results - - - - - - - - - -
update-motd PASS
autopkgtest [13:50:21]: @@@@@@@@@@@@@@@@@@@@ summary
show-motd PASS
update-motd PASS
```

[Utkarsh Gupta (utkarsh)]

For posterity, pasting my logs here:
https://paste.ubuntu.com/p/vPn6V64TxF/

For me, it passes in Hirsute and Impish image but fails in container. Can't reproduce this for amd64, at least.

[Christian Ehrhardt  (paelzer)]

I can confirm that as of now it runs fine against
- impish
- impish + UA from proposed
- imposh proposed

The test installs all users of update-motd

It does not list ubuntu-advantage-tools but since they are in the image one would think that it still would be there.

The test intention is:
1. run update-motd while ALL motd scripts installed
2. ensure all are installed

---

To do so for #1 it has the above mentioned dependencies and then runs "update-motd".
This works fine.

---

#2 is checked via "apt-file" to deliver ALL that exist in the archive
And then followed by a check if all those are present on the current system.

The former advantage-tools 26.2 had no motd scripts.
The new 27.0.1 has two:
root@h:~# dpkg -L ubuntu-advantage-tools | grep motd
/etc/update-motd.d
/etc/update-motd.d/88-esm-announce
/etc/update-motd.d/91-contract-ua-esm-status

Those two are correctly reported in the "apt-file" output of impish.

root@h:~# apt-file search /etc/update-motd.d | grep adv
ubuntu-advantage-tools: /etc/update-motd.d/88-esm-announce
ubuntu-advantage-tools: /etc/update-motd.d/91-contract-ua-esm-status

But, since the test runs on "autopkgtest image + new package install" but NOT "apt upgrade" it isn't really testing with 27.0.1.

Most of us usually run autopkgtest with -U|--apt-upgrade and therefore we couldn't reproduce.
But with the following it does show up in a local test:

$ sudo ~/work/autopkgtest/autopkgtest/runner/autopkgtest --no-built-binaries --apt-pocket=proposed=src:ubuntu-advantage-tools --shell-fail update-motd_3.7.dsc -- qemu --ram-size=1536 --cpus 2 ~/work/autopkgtest-impish-amd64.img
...
ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/88-esm-announce: /etc/update-motd.d/88-esm-announce
ERROR: Untested update-motd snippet: ubuntu-advantage-tools: /etc/update-motd.d/91-contract-ua-esm-status: /etc/update-motd.d/91-contract-ua-esm-status

This is how the broken testbed looks like:
ubuntu@autopkgtest:~$ apt-cache policy ubuntu-advantage-tools
ubuntu-advantage-tools:
  Installed: 26.2
  Candidate: 27.0.1
  Version table:
     27.0.1 995
        500 http://archive.ubuntu.com/ubuntu impish-proposed/main amd64 Packages
 *** 26.2 990
        990 http://archive.ubuntu.com/ubuntu impish/main amd64 Packages
        100 /var/lib/dpkg/status

Options that come to mind
a) change update-motd d/t/control to install ubuntu-advantage-tools which will update it
   => does not work, it is considered already there and not upgraded
What is left is
b) wait until an image is re-spun (not this will also affect all SRUed releases at some point)
c) change update-motd d/t/update-motd to call a full "apt upgrade -y" before the test

(c) does not seems super charming to SRU as test-only fix :-/
I wonder if it would be painful but better to get the images respinned fast as - after all - it is a false positive that will go away.

[Balint Reczey (rbalint)]

@paelzer IMO Ubuntu CI not upgrading all packages (-U) by default is a bug because it tests a partially upgraded system (which is not supported) and may even test packages which are not present anymore in the archive (because they are superseded by a newer version).

[Iain Lane (laney)]

> IMO Ubuntu CI not upgrading all packages (-U) by default is a bug because it tests a partially upgraded system

What do you mean please? I can see in the linked log "--apt-upgrade" in the commandline right at the top. Can you please check this assertion?

The mentioned ubuntu-advantage-tools is currently only in impish-proposed. What's happening here is the `apt-file search` which update-motd does as part of its test finds update-motd snippets in any enabled pocket, and that includes proposed even for packages we aren't taking from there.

The new snippets in ubuntu-advantage-tools in proposed are found, not tested (because they are not in the test-depends [would need a >= version to get the right one, as the package is not in release atm, and maybe a manual trigger on ubuntu-advantage-tools, not sure if proposed-migration would be smart enough to figure that out]), and this is making the error happen.

So, a buggy test in update-motd. I think the idea of this `update-motd` test is a bit questionable - it's not really update-motd's job to test everything which integrates with it works all of the time. It should probably look more like an `autodep8` test in which the integrations test themselves .

[Balint Reczey (rbalint)]

My mistake, Ubuntu CI actually passes --apt-upgrade to autopkgtest.

[Balint Reczey (rbalint)]

@paelzer I've uploaded update-motd adding ubuntu-advantage-tools as a test dependency, because it is accidental that it is already installed on test images.

The apt-file check is not ideal, because it lists packages which are in -proposed but pinning does not allow them to be installed. As a mitigation I've retried all affected tests with ubuntu-advantage-tools also added as a trigger. The apt-file results could be filtered by taking pinning
into consideration, but this would not be perfect easily either, thus I think it is not worth it.

Yes, SRU-ing a test-only fix is not ideal either. The tests are there to ensure that update-motd scripts can be run as a normal user which is done on WSL: https://balintreczey.hu/blog/introducing-show-motd-message-of-the-day-for-wsl-and-container-shells/ . An alternative of testing every script by update-motd would be generating similar tests for every package shipping and update-motd script.

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-motd - 3.8

---------------
update-motd (3.8) impish; urgency=medium

  * debian/tests/control: Test ubuntu-advantage-tools scripts, too
    (LP: #1926660)

 -- Balint Reczey <email address hidden> Thu, 06 May 2021 17:42:45 +0200

[Balint Reczey (rbalint)]

@laney Sorry, I have not seen your comment #8 before writing mine. I did consider writing an autodep8 test, but it seemed to be overkill due the small amount of packages to use it.
Now with the appearance of this bug I've reconsidered that and autodep8 would be better to ensure good coverage than the apt-file based check.