autopkgtest package "lxd" test has started failing

[Dan Streetman]

[impact]

"lxd launch" behavior recently changed to parse any queued input on stdin and process it as a yaml file.

Since the "autopkgtest-build-lxd" script calls "lxd launch", and input queued to stdin when calling "autopkgtest-build-lxd" will cause the internal call to lxd to fail, such as calling "autopkgtest-build-lxd" from inside a here document (see test case below).

the failure is seen in (at least) the "lxd" autopkgtest from the autopkgtest package itself, e.g.:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco/disco/amd64/a/autopkgtest/20190923_160817_66dbe@/log.gz

[test case]

root@autopkgtest:~# cat <<EOF | su - ubuntu
> autopkgtest-build-lxd ubuntu:bionic
> echo ok done
> EOF
Error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `echo ok...` into api.ContainerPut

run the autopkgtests for the package 'autopkgtest', or check autopkgtest.ubuntu.com, e.g.:
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/disco/amd64
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/disco/i386
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/eoan/amd64
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/eoan/i386

the autopkgtest test case is failing only for disco and eoan, and only for amd64/i386 as the 'lxd' test is skipped for non-intel archs.

the autopkgtest test case does not fail on bionic or xenial, because the tests there run with the lxd deb, which is still at the older version.
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/bionic/amd64

However, the lxd snap can be installed in xenial or bionic, and the failure is reproducable when the lxd snap is installed, i.e.:

$ sudo apt remove lxd
$ sudo snap install lxd
...perform test case

[regression potential]

this only redirects stdin from /dev/null, for the call to 'lxc launch' inside autopkgtest-build-lxd; so the regression potential should be low. Any regressions would almost certainly involve a failure during the call to autopkgtest-build-lxd, during the creation of the lxd container.

[other info]

this is reproducable with the current packages from disco-updates using a local qemu vm for testing.

note that the autopkgtest fails only for disco and eoan, becuase on xenial and bionic the lxd deb is used, which still has the older behavior, however the lxd snap is available to install on both xenial and bionic, so this problem still exists for both those releases, as well as disco and eoan.

also, this is not just a testcase failure - this is actual new behavior that can break existing users of the "autopkgtest-build-lxd" script in the manner described in the test case.

I put the 'importance' of this as low because this bug will only be reproduced when calling autopkgtest-build-lxd from inside a here document passed to a shell, or otherwise called with input queued on stdin, which seems like an unusual way to call autopkgest-build-lxd, although still a perfectly valid way to use it.

[Stéphane Graber (stgraber)]

This is because the test is somehow passing data to "lxc init" or "lxc launch" through stdin.

Up until LXD 3.17, we'd only consume stdin in some rare cases during init/launch. But now we're using it as a way to pre-seed the container's YAML configuration, as the data fed through stdin isn't a YAML document, this fails.

autopkgtest should be changed to either call those commands in a way that doesn't provide them with invalid stdin input or by simply using "< /dev/null" to ensure the particular command does not inherit whatever stdin currently contains.

[Dan Streetman (ddstreet)]

ah, ok, thanks @stgraber!

This will indeed need some surgery to autopkgtest, and might start causing more widespread failures since the autopkgtest LxdRunner itself appears to be at fault.

[Dan Streetman (ddstreet)]

> might start causing more widespread failures since the autopkgtest LxdRunner itself appears to be at fault.

bah, those are classes in autopkgtest's self-tests, so should only affect autopkgtest's tests. still needs fixing, but not urgent.

[Dan Streetman (ddstreet)]

@stgraber, just to clarify, calls to lxd like this now fail:

root@autopkgtest:~# cat <<EOF | su - ubuntu
> lxc launch ubuntu:bionic
> echo ok done
> EOF
Error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `echo ok...` into api.ContainerPut

is it really expected that lxd should fail when used with input queued on stdin like this?

[Dan Streetman (ddstreet)]

fixed upstream in Debian:
https://salsa.debian.org/ci-team/autopkgtest/commit/747d55377c943143f71a56c6decb8e26957f178d

[Steve Langasek (vorlon)]

Hello Dan, or anyone else affected,

Accepted autopkgtest into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.10ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 5.10ubuntu2

---------------
autopkgtest (5.10ubuntu2) eoan; urgency=medium

  * tools/autopkgtest-build-lxd: pass /dev/null on stdin to lxc launch
    (LP: #1845037)

 -- Dan Streetman <email address hidden> Thu, 26 Sep 2019 08:48:21 -0400

[Łukasz Zemczak (sil2100)]

Hello Dan, or anyone else affected,

Accepted autopkgtest into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.3.1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Dan, or anyone else affected,

Accepted autopkgtest into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/3.20.4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted autopkgtest (5.3.1ubuntu1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

libnftnl/unknown (ppc64el, amd64, arm64, armhf, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted autopkgtest (3.20.4ubuntu1.1) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

autopkgtest/3.20.4ubuntu1.1 (armhf)
debci/1.0.2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

autopkgtest failures:

autopkgtest (xenial/armhf): bug 1847242, ignore

also i'm re-running several other failed tests

[Dan Streetman (ddstreet)]

The 'lxd' autopkgtest now passes on all archs/releases.

[Dan Streetman (ddstreet)]

only autopkgtest failures left can be ignored:

autopkgtest (xenial/armhf): bug 1847242, ignore
debci (xenial/arm64): flaky, test usually fails, ignore.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted autopkgtest (3.20.4ubuntu1.1) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

autopkgtest/3.20.4ubuntu1.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!