autopkgtest package "lxd" test has started failing

Bug #1845037 reported by Dan Streetman on 2019-09-23
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
autopkgtest (Ubuntu)
Low
Dan Streetman
Xenial
Low
Dan Streetman
Bionic
Low
Dan Streetman
Disco
Low
Dan Streetman
Eoan
Low
Dan Streetman

Bug Description

[impact]

"lxd launch" behavior recently changed to parse any queued input on stdin and process it as a yaml file.

Since the "autopkgtest-build-lxd" script calls "lxd launch", and input queued to stdin when calling "autopkgtest-build-lxd" will cause the internal call to lxd to fail, such as calling "autopkgtest-build-lxd" from inside a here document (see test case below).

the failure is seen in (at least) the "lxd" autopkgtest from the autopkgtest package itself, e.g.:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco/disco/amd64/a/autopkgtest/20190923_160817_66dbe@/log.gz

[test case]

root@autopkgtest:~# cat <<EOF | su - ubuntu
> autopkgtest-build-lxd ubuntu:bionic
> echo ok done
> EOF
Error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `echo ok...` into api.ContainerPut

run the autopkgtests for the package 'autopkgtest', or check autopkgtest.ubuntu.com, e.g.:
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/disco/amd64
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/disco/i386
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/eoan/amd64
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/eoan/i386

the autopkgtest test case is failing only for disco and eoan, and only for amd64/i386 as the 'lxd' test is skipped for non-intel archs.

the autopkgtest test case does not fail on bionic or xenial, because the tests there run with the lxd deb, which is still at the older version.
http://autopkgtest.ubuntu.com/packages/a/autopkgtest/bionic/amd64

However, the lxd snap can be installed in xenial or bionic, and the failure is reproducable when the lxd snap is installed, i.e.:

$ sudo apt remove lxd
$ sudo snap install lxd
...perform test case

[regression potential]

this only redirects stdin from /dev/null, for the call to 'lxc launch' inside autopkgtest-build-lxd; so the regression potential should be low. Any regressions would almost certainly involve a failure during the call to autopkgtest-build-lxd, during the creation of the lxd container.

[other info]

this is reproducable with the current packages from disco-updates using a local qemu vm for testing.

note that the autopkgtest fails only for disco and eoan, becuase on xenial and bionic the lxd deb is used, which still has the older behavior, however the lxd snap is available to install on both xenial and bionic, so this problem still exists for both those releases, as well as disco and eoan.

also, this is not just a testcase failure - this is actual new behavior that can break existing users of the "autopkgtest-build-lxd" script in the manner described in the test case.

I put the 'importance' of this as low because this bug will only be reproduced when calling autopkgtest-build-lxd from inside a here document passed to a shell, or otherwise called with input queued on stdin, which seems like an unusual way to call autopkgest-build-lxd, although still a perfectly valid way to use it.

Stéphane Graber (stgraber) wrote :

This is because the test is somehow passing data to "lxc init" or "lxc launch" through stdin.

Up until LXD 3.17, we'd only consume stdin in some rare cases during init/launch. But now we're using it as a way to pre-seed the container's YAML configuration, as the data fed through stdin isn't a YAML document, this fails.

autopkgtest should be changed to either call those commands in a way that doesn't provide them with invalid stdin input or by simply using "< /dev/null" to ensure the particular command does not inherit whatever stdin currently contains.

affects: lxd (Ubuntu) → autopkgtest (Ubuntu)
Dan Streetman (ddstreet) wrote :

ah, ok, thanks @stgraber!

This will indeed need some surgery to autopkgtest, and might start causing more widespread failures since the autopkgtest LxdRunner itself appears to be at fault.

Changed in autopkgtest (Ubuntu):
assignee: nobody → Dan Streetman (ddstreet)
importance: Undecided → High
status: New → In Progress
Dan Streetman (ddstreet) wrote :

> might start causing more widespread failures since the autopkgtest LxdRunner itself appears to be at fault.

bah, those are classes in autopkgtest's self-tests, so should only affect autopkgtest's tests. still needs fixing, but not urgent.

Changed in autopkgtest (Ubuntu Eoan):
importance: High → Low
Changed in autopkgtest (Ubuntu Disco):
importance: Undecided → Low
Changed in autopkgtest (Ubuntu Bionic):
importance: Undecided → Low
Changed in autopkgtest (Ubuntu Xenial):
importance: Undecided → Low
Changed in autopkgtest (Ubuntu Disco):
status: New → In Progress
Changed in autopkgtest (Ubuntu Bionic):
status: New → In Progress
Changed in autopkgtest (Ubuntu Xenial):
status: New → In Progress
Changed in autopkgtest (Ubuntu Disco):
assignee: nobody → Dan Streetman (ddstreet)
Changed in autopkgtest (Ubuntu Bionic):
assignee: nobody → Dan Streetman (ddstreet)
Changed in autopkgtest (Ubuntu Xenial):
assignee: nobody → Dan Streetman (ddstreet)
Dan Streetman (ddstreet) wrote :

@stgraber, just to clarify, calls to lxd like this now fail:

root@autopkgtest:~# cat <<EOF | su - ubuntu
> lxc launch ubuntu:bionic
> echo ok done
> EOF
Error: yaml: unmarshal errors:
  line 1: cannot unmarshal !!str `echo ok...` into api.ContainerPut

is it really expected that lxd should fail when used with input queued on stdin like this?

Dan Streetman (ddstreet) on 2019-09-25
description: updated
Dan Streetman (ddstreet) on 2019-09-26
description: updated
Dan Streetman (ddstreet) on 2019-09-27
description: updated
description: updated

Hello Dan, or anyone else affected,

Accepted autopkgtest into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.10ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: block-proposed-disco
Changed in autopkgtest (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco
Changed in autopkgtest (Ubuntu Eoan):
status: In Progress → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autopkgtest - 5.10ubuntu2

---------------
autopkgtest (5.10ubuntu2) eoan; urgency=medium

  * tools/autopkgtest-build-lxd: pass /dev/null on stdin to lxc launch
    (LP: #1845037)

 -- Dan Streetman <email address hidden> Thu, 26 Sep 2019 08:48:21 -0400

Changed in autopkgtest (Ubuntu Eoan):
status: Fix Committed → Fix Released
Łukasz Zemczak (sil2100) wrote :

Hello Dan, or anyone else affected,

Accepted autopkgtest into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.3.1ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in autopkgtest (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Changed in autopkgtest (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Łukasz Zemczak (sil2100) wrote :

Hello Dan, or anyone else affected,

Accepted autopkgtest into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/3.20.4ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted autopkgtest (5.3.1ubuntu1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

libnftnl/unknown (ppc64el, amd64, arm64, armhf, i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted autopkgtest (3.20.4ubuntu1.1) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

autopkgtest/3.20.4ubuntu1.1 (armhf)
debci/1.0.2 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Dan Streetman (ddstreet) wrote :

autopkgtest failures:

autopkgtest (xenial/armhf): bug 1847242, ignore

also i'm re-running several other failed tests

Dan Streetman (ddstreet) wrote :

The 'lxd' autopkgtest now passes on all archs/releases.

tags: added: verification-done verification-done-bionic verification-done-disco verification-done-xenial
removed: verification-needed verification-needed-bionic verification-needed-disco verification-needed-xenial
Dan Streetman (ddstreet) wrote :

only autopkgtest failures left can be ignored:

autopkgtest (xenial/armhf): bug 1847242, ignore
debci (xenial/arm64): flaky, test usually fails, ignore.

tags: added: block-proposed-bionic block-proposed-xenial
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers