curtin does not retry when gpg fails to recv key

[Ryan Harper]

==== Begin SRU Template ========
[Impact]
Curtin installation can be configured to install additional apt
repositories and import gpg keys from a remote keyserver by fingerprint.
'gpg --recv' can transiently fail, causing installation failure.

The fix is to just retry
  gpg --keyserver keyserver.ubuntu.com --recv <key>

[Test Case]
Run curtin vmtest on XenialTestAptSrcDisablePockets
 ./tools/jenkins-runner tests/vmtests/test_apt_source.py

The named test pushes curtin through use of gpg --keyserver.
It does not guarantee failure and re-try, but does test the
happy path. Even if the gpg server was available on the first
time, we at very least then verify that there is no regression.

[Regression Potential]
Low risk of regression. The code change is just to sleep and retry
a command if it failed.

[Other Info]
When looking further into some failures in add-apt-repository that were
not fixed by re-trying, we found bug 1532855 . It is not explicitly
related to this bug, but is a similar failure path.

==== End SRU Template ========

In some cases we have transient network failure while attempting to recv gpg keys when configuring apt.

https://jenkins.ubuntu.com/server/job/curtin-vmtest/737/artifact/output/XenialTestAptSrcDisablePockets/logs/install-serial.log

[ 40.161264] cloud-init[1528]: Command: ['gpg', '--keyserver', 'keyserver.ubuntu.com', '--recv', '0E72 9061 0D2F 6DC4 D65E A921 9A31 4EC5 F470 A0AC']
[ 40.162213] cloud-init[1528]: Exit code: 2
[ 40.163711] cloud-init[1528]: Reason: -
[ 40.164256] cloud-init[1528]: Stdout: "gpgkeys: key 0E7290610D2F6DC4D65EA9219A314EC5F470A0AC can't be retrieved\n"
[ 40.174200] cloud-init[1528]: Stderr: 'gpg: requesting key F470A0AC from hkp server keyserver.ubuntu.com\ngpg: no valid OpenPGP data found.\ngpg: Total number processed: 0\ngpg: keyserver communications error: keyserver helper general error\ngpg: keyserver communications error: unknown pubkey algorithm\ngpg: keyserver receive failed: unknown pubkey algorithm\n'
[ 40.180654] cloud-init[1528]: During handling of the above exception, another exception occurred:
[ 40.182056] cloud-init[1528]: Traceback (most recent call last):
[ 40.183644] cloud-init[1528]: File "/curtin/curtin/gpg.py", line 65, in getkeybyid
[ 40.185067] cloud-init[1528]: recv_key(keyid, keyserver=keyserver)
[ 40.186653] cloud-init[1528]: File "/curtin/curtin/gpg.py", line 48, in recv_key
[ 40.187895] cloud-init[1528]: (key, keyserver, error))

Curtin should retry this command.

[Ryan Harper (raharper)]

Suggested fix:

=== modified file 'curtin/gpg.py'
--- curtin/gpg.py 2016-06-23 15:04:21 +0000
+++ curtin/gpg.py 2017-02-02 17:28:22 +0000
@@ -41,7 +41,7 @@
     LOG.debug('Receive gpg key "%s"', key)
     try:
         util.subp(["gpg", "--keyserver", keyserver, "--recv", key],
- capture=True)
+ capture=True, retries=(1, 2, 5, 10))
     except util.ProcessExecutionError as error:
         raise ValueError(('Failed to import key "%s" '
                           'from server "%s" - error %s') %

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted curtin into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr460-0ubuntu1~16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Ryan, or anyone else affected,

Accepted curtin into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/curtin/0.1.0~bzr460-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

I've run curtin's vmtest testsuite with the test provided in the
description by doing the following.

# get curtin for it tests at revision 460
$ bzr branch lp:curtin curtin.dist
$ bzr branch -r 460 curtin.dist curtin-r460
$ cd curtin-r460
## need to get 'curtainer' and 'curtin-in-container' from trunk
$ cp ../trunk.dist/tools/{curtainer,curtin-from-container} tools/
$ ./tools/vmtest-system-setup
$ ./tools/curtainer --proposed xenial sm-curtin-x1
...
Unpacking curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up curtin-common (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up python3-curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...
Setting up curtin (0.1.0~bzr460-0ubuntu1~16.04.1) ...

# bug 1656369 tests/vmtests/test_multipath.py
# bug 1659509 tests/unittests/test_commands_block_meta.py
# bug 1661337 tests/vmtests/test_apt_source.py
$ name=sm-curtin-x1
$ CURTIN_VMTEST_TOPDIR=$PWD/$name CURTIN_VMTEST_CURTIN_EXE="./tools/curtin-from-container $name curtin" \
   ./tools/jenkins-runner \
     tests/vmtests/test_multipath.py \
     tests/unittests/test_commands_block_meta.py \
     tests/vmtests/test_apt_source.py

I'm attaching a tarball of the output directory sm-curtin-x1

Note, due to packaging bug 1666986, we see in the installation logs:
   curtin: Installation started. (0.1.0)
when that is fixed, we'll start seeing 0.1.0~bzr460-0ubuntu1~16.04.1.

[Scott Moser (smoser)]

I've run curtin's vmtest testsuite with the test provided in the
description by doing the following.

# get curtin for it tests at revision 460
$ bzr branch lp:curtin curtin.dist
$ bzr branch -r 460 curtin.dist curtin-r460
$ cd curtin-r460
## need to get 'curtainer' and 'curtin-in-container' from trunk
$ cp ../trunk.dist/tools/{curtainer,curtin-from-container} tools/
$ ./tools/vmtest-system-setup
$ ./tools/curtainer images:ubuntu/yakkety --proposed sm-curtin-y1
# note, used images: due to bug 1668710
$ ./tools/curtainer --proposed yakkety sm-curtin-y1
....
Setting up curtin (0.1.0~bzr460-0ubuntu1~16.10.1) ...

# bug 1656369 tests/vmtests/test_multipath.py
# bug 1659509 tests/unittests/test_commands_block_meta.py
# bug 1661337 tests/vmtests/test_apt_source.py
$ name=sm-curtin-y1
$ CURTIN_VMTEST_TOPDIR=$PWD/$name CURTIN_VMTEST_CURTIN_EXE="./tools/curtin-from-container $name curtin" \
   ./tools/jenkins-runner \
     tests/vmtests/test_multipath.py \
     tests/unittests/test_commands_block_meta.py \
     tests/vmtests/test_apt_source.py

Note, due to packaging bug 1666986, we see in the installation logs:
   curtin: Installation started. (0.1.0)
when that is fixed, we'll start seeing 0.1.0~bzr460-0ubuntu1~16.10.1.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr460-0ubuntu1~16.10.1

---------------
curtin (0.1.0~bzr460-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * New upstream snapshot.
    - Install zipl in target on s390x arch. (LP: #1662346)
    - avoid UnicodeDecode error on passing non-utf8 into shlex
    - adjustments to version string handling, improved pack unit tests.
    - helpers/common: Add grub install debugging output
    - curtin: add version module and display in output and logs
    - content decoding in load_file, apply_net raise exception on errors
    - gpg: retry when recv'ing gpg keys fail (LP: #1661337)
    - Add clear_holders checks to disk and partition handlers (LP: #1659509)
    - net: add new lines after rendered static routes. (LP: #1649652)
    - multipath: don't run update-grub; setup_grub will handle this better.
      (LP: #1656369)
    - Test changes:
      - vmtest: Add tests for zesty and Trusty HWE-X kernels.
      - tests: fix tox tip-pycodestyle complaints
      - image-sync: add debugging output to help diagnose errors
      - vmtest: change get_curtin_version to use version subcommand.
      - Remove style checking during build and add latest style checks to tox
      - subp doc an unit test improvements.
      - flake8: remove unused variable.
      - vmtest: Add the ability to add extra config files to test execution.
      - vmtest: overhaul image sync
      - vmtest: skip apt-proxy test if not set
      - vmtest: add 'webserv' helper
      - vmtest: add CURTIN_VMTEST_CURTIN_EXE variable.

 -- Scott Moser <email address hidden> Thu, 16 Feb 2017 22:30:13 -0500

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for curtin has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package curtin - 0.1.0~bzr460-0ubuntu1~16.04.1

---------------
curtin (0.1.0~bzr460-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * New upstream snapshot.
    - Install zipl in target on s390x arch. (LP: #1662346)
    - avoid UnicodeDecode error on passing non-utf8 into shlex
    - adjustments to version string handling, improved pack unit tests.
    - helpers/common: Add grub install debugging output
    - curtin: add version module and display in output and logs
    - content decoding in load_file, apply_net raise exception on errors
    - gpg: retry when recv'ing gpg keys fail (LP: #1661337)
    - Add clear_holders checks to disk and partition handlers (LP: #1659509)
    - net: add new lines after rendered static routes. (LP: #1649652)
    - multipath: don't run update-grub; setup_grub will handle this better.
      (LP: #1656369)
    - Test changes:
      - vmtest: Add tests for zesty and Trusty HWE-X kernels.
      - tests: fix tox tip-pycodestyle complaints
      - image-sync: add debugging output to help diagnose errors
      - vmtest: change get_curtin_version to use version subcommand.
      - Remove style checking during build and add latest style checks to tox
      - subp doc an unit test improvements.
      - flake8: remove unused variable.
      - vmtest: Add the ability to add extra config files to test execution.
      - vmtest: overhaul image sync
      - vmtest: skip apt-proxy test if not set
      - vmtest: add 'webserv' helper
      - vmtest: add CURTIN_VMTEST_CURTIN_EXE variable.

 -- Scott Moser <email address hidden> Wed, 08 Feb 2017 19:40:38 -0500

[Scott Moser (smoser)]

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.