[cloud-archive] GPG signature invalid: BADSIG

[Rafael Folco]

Summary
=======
UCA returns GPG error (BADSIG) on minute 50-59 (fifity-something), so it fails to install "unauthenticated" packages.

There might be a cron job running on UCA repo within 50-59 min of each hour? Or perhaps a maintenance script that is causing GPG keys to be invalid during that short time ?

This is OK when running manually, so you can retry minutes later and it works. However, It impacts OpenStack CI, which runs 24x7 per-patch basis jobs, in an automated and atomically way.

Note: We observed that this happens always in the minute 50-59, and has not happened in a minute out of this range (0-49).

Note2: This could be reproduced out of our labs (At Unicamp's Mini cloud for example), in a totally different network.

Note3: Allowing unauthenticated packages is not desired.

Arch=ppc64le
Ubuntu=Xenial
UCA=Ocata

Steps to reproduce
==================
- On a ppc64le machine (Power8), running xenial
- at min 50-59 (fifty-something), add UCA repo (Ubuntu Cloud Archive)
$ sudo add-apt-repository -y cloud-archive:ocata
- Update apt repos
$ sudo apt-get update
- GPG error (BADSIG) is seen
GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release: The following signatures were invalid: BADSIG 5EDB1B62EC4926EA Canonical Cloud Archive Signing Key <email address hidden>
- install openvswitch-switch
$ sudo apt-get install openvswitch-switch
E: There were unauthenticated packages and -y was used without --allow-unauthenticated

Output
======
2017-05-25 16:50:47.324 | ++ functions-common:apt_get_update:1050 : timeout 300 sh -c 'while ! sudo http_proxy= https_proxy= no_proxy= apt-get update; do sleep 30; done'
2017-05-25 16:50:47.551 | Ign:1 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata InRelease
2017-05-25 16:50:47.561 | Hit:2 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
2017-05-25 16:50:47.639 | Get:3 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release [7882 B]
2017-05-25 16:50:47.643 | Get:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release.gpg [543 B]
2017-05-25 16:50:47.652 | Hit:5 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease
2017-05-25 16:50:47.742 | Hit:6 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease
2017-05-25 16:50:47.824 | Ign:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release.gpg
2017-05-25 16:50:47.835 | Hit:7 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease
2017-05-25 16:50:47.916 | Get:8 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata/main ppc64el Packages [145 kB]
2017-05-25 16:50:47.990 | Fetched 154 kB in 0s (240 kB/s)
2017-05-25 16:50:48.647 | Reading package lists...
2017-05-25 16:50:48.676 | W: GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release: The following signatures were invalid: BADSIG 5EDB1B62EC4926EA Canonical Cloud Archive Signing Key <email address hidden>
2017-05-25 16:50:48.676 | W: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release' is not signed.

...

2017-05-25 16:51:50.654 | + functions-common:real_install_package:1263 : apt_get install fakeroot make openvswitch-switch
2017-05-25 16:51:50.672 | + functions-common:apt_get:1076 : sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install fakeroot make openvswitch-switch
2017-05-25 16:51:50.709 | Reading package lists...
2017-05-25 16:51:50.834 | Building dependency tree...
2017-05-25 16:51:50.835 | Reading state information...
2017-05-25 16:51:50.934 | fakeroot is already the newest version (1.20.2-1ubuntu1).
2017-05-25 16:51:50.934 | fakeroot set to manually installed.
2017-05-25 16:51:50.934 | make is already the newest version (4.1-6).
2017-05-25 16:51:50.934 | The following NEW packages will be installed:
2017-05-25 16:51:50.934 | openvswitch-common openvswitch-switch python-six
2017-05-25 16:51:50.946 | 0 upgraded, 3 newly installed, 0 to remove and 14 not upgraded.
2017-05-25 16:51:50.946 | Need to get 2047 kB of archives.
2017-05-25 16:51:50.946 | After this operation, 12.0 MB of additional disk space will be used.
2017-05-25 16:51:50.946 | WARNING: The following packages cannot be authenticated!
2017-05-25 16:51:50.946 | openvswitch-common openvswitch-switch
2017-05-25 16:51:50.947 | E: There were unauthenticated packages and -y was used without --allow-unauthenticated

Logs taken from:
http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/67/465767/5/check/tempest-dsvm-full-xenial/fcf1cea/devstacklog.txt.gz
***This log expires

[Rafael Folco (rafaelfolco)]

This continues to happen in the same way as described above. This is a showstopper for CI jobs running in an automated way.

2017-06-19 13:53:22.419 | W: GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release: The following signatures were invalid: BADSIG 5EDB1B62EC4926EA Canonical Cloud Archive Signing Key <email address hidden>
2017-06-19 13:53:22.420 | W: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release' is not signed.

[Manoj Iyer (manjo)]

Could you please do:

$ sudo add-apt-repository -y cloud-archive:ocata
$ sudo apt install ubuntu-cloud-keyring
$ sudo apt update
$ sudo apt install openvswitch-switch

Let me know if that works.

[Rafael Folco (rafaelfolco)]

Hi Manoj,

The first command implies in installing ubuntu-cloud-keyring already, so if you try to re-install it, you'll get:
ubuntu-cloud-keyring is already the newest version (2012.08.14).

This is what happens in the minute 51 of any hour, for example:

$ date; sudo ./uca.sh | tee uca.log; date
Wed Jun 21 12:51:01 UTC 2017
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  ubuntu-cloud-keyring
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 5086 B of archives.
After this operation, 34.8 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial/universe ppc64el ubuntu-cloud-keyring all 2012.08.14 [5086 B]
Fetched 5086 B in 0s (91.2 kB/s)
                                Selecting previously unselected package ubuntu-cloud-keyring.
(Reading database ... 83972 files and directories currently installed.)
Preparing to unpack .../ubuntu-cloud-keyring_2012.08.14_all.deb ...
Unpacking ubuntu-cloud-keyring (2012.08.14) ...
Setting up ubuntu-cloud-keyring (2012.08.14) ...
Importing ubuntu-cloud.archive.canonical.com keyring
OK
Processing ubuntu-cloud.archive.canonical.com removal keyring
gpg: /etc/apt/trustdb.gpg: trustdb created
OK
Reading package lists...
Building dependency tree...
Reading state information...
ubuntu-cloud-keyring is already the newest version (2012.08.14).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Ign:1 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
Get:3 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release [7882 B]
Get:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release.gpg [543 B]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [102 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [102 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [102 kB]
Ign:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main ppc64el Packages
Ign:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe ppc64el Packages
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main ppc64el Packages [489 kB]
Ign:4 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release.gpg
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe ppc64el Packages [426 kB]
Ign:10 http://ports.ubuntu.com/ubuntu-ports xenial-backports/universe ppc64el Packages
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-backports/universe ppc64el Packages [5256 B]
Get:11 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata/main ppc64el Packages [145 kB]
Fetched 1381 kB in 1s (747 kB/s)
Reading package lists...
W: GPG error: http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release: The following signatures were invalid: BADSIG 5EDB1B62EC4926EA Canonical Cloud Archive Signing Key <email address hidden>
W: The repository 'http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata Release' is not signed.
Reading package lists.....

[Kyle L. Henderson (kyleh)]

This UCA issue has been documented here also: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1657440

The comments by smatzek are particularly thorough.

[Corey Bryant (corey.bryant)]

Hello Rafael,

Thank you for reporting this bug.

Following is a short summary of what bug #1657440 describes.

There's an hourly window when the UCA is being updated. An apt update call during this window causes the Release file to be downloaded and Release.gpg file to not be downloaded. The missing Release.gpg file causes the "unauthenticated" packages.

According to bug #1657440 this issue has been partially fixed back to xenial. The missing Release.gpg file issue still exists, however, prior to the fix to apt in bug #1657440, the missing Release.gpg issue would not correct until an hour had passed and another apt update was run. At this point, an ensuing apt update call apparently resolves the issue.

So bottom line, there is still an issue but it is at least resolvable with an ensuing apt update.

There are scripts to recreate in bug #1657440.

And here is the patch to apt for reference: https://github.com/Debian/apt/commit/2a6d2e9c0781a0a7bb5d2aad7b8bdbee315d4461

[Sahid Orentino (sahid-ferdjaoui)]

I've marked the issue as "Incomplete" since we don"t have fresh news from reporter and based on comment #5 the issue should now be fixed.

Rafael, please feel free to mark it back to "New" if you are still suffering an issue.

[Chris MacNaughton (chris.macnaughton)]

I believe that this has also been finally resolved with https://bugs.launchpad.net/cloud-archive/+bug/1772060