autopkgtest fail in version 14.2 on test-display-streaming

Bug #1834286 reported by Christian Ehrhardt  on 2019-06-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
spice-vdi
New
Unknown
spice (Ubuntu)
High
Unassigned

Bug Description

After the merge of 14.2 I realized that the autopkgtests failed.

There are a few changes required that are due to the new version.
    - d/t/automated-tests: spice-common moved into dir subprojects
    - d/t/automated-tests: option --enable-automated-tests now is always on"
    - d/t/automated-tests, d/t/control: make tests more debuggable by allowing
      stderr
    - d/t/control: install new test dependency python-pil
    - d/t/base_test.ppm, d/t/regression-test.py: provide test resources from
      upstream git not part of the released tarball anymore
    - d/source/include-binaries: allow binary base_test.ppm in package

Of these particularly interesting is that the python script and the image files are not even part of the release tarball anymore. So I wonder if this is really supposed to still be run that way.

I got it resolved to "run" the test but it complained about an image mismatch.
[...]
req_cursor_notification
wrote screen shot to output.ppm
FAIL: Regressions were found!
      Please, take a look in your code and go fix it!
(process:21472): Spice-WARNING **: 08:00:30.990: Connection reset by peer
(process:21472): Spice-WARNING **: 08:00:31.136: Connection reset by peer
set_client_capabilities: present 0 caps 0
client_monitors_config: NULL monitors_config

There were plenty of changes to this test to make it work on windows, I wonder if one of them might have broken it - at least in the way we used to execute it.

The images it generates are reproducible, but don't match.

Since I had to tweak quite a lot to get it running in the first place I'm not entirely sure the test is valid at the moment.

See comment #3 below for details on the old behavior which also was incomplete running the test but not comparing output at all.

Therefore I'll upload the test fixes to run them (but it will ignore this error for now) - which is exactly the old state.
=> https://code.launchpad.net/~paelzer/ubuntu/+source/spice/+git/spice/+merge/369300

To eventually resolve this I will report the problem upstream and depending on the outcome of that discussion we can fix our usage of the test OR drop it completely if it is no more meant to be run that way.

expected test file from git show upstream/master:server/tests/base_test.ppm

mismatching output created by the tests

Actually I realuzed that the old test 0.14.0-1ubuntu5 is working and also had no regression-test.py nor the ppm files.
But the code in server/tests/test-display-base.c calls that python helper just the same:
  execve("regression-test.py", argv, envp);
So why isn't that failing, maybe it was never a valid test to begin with?.

I entered a test env of the test on Disco (with these issues present) and checked how/why the test complete there.

It has an RC of 0, but that it has always if not crashing.
Still no files around:
ubuntu@autopkgtest:/tmp/autopkgtest.xBVfjt/build.efW/src$ find -name '*.ppm*'
ubuntu@autopkgtest:/tmp/autopkgtest.xBVfjt/build.efW/src$ find -name '*regression*'

Function regression_test is hit often.
Thread 1 "test-display-st" hit Breakpoint 1, regression_test () at test-display-base.c:101
101 {

There a child is supposed to start the test:
 114 pid = fork();
 115 if (pid == 0) {
 116 char buf[PATH_MAX];
 117 char *argv[] = { NULL };
 118 char *envp[] = {buf, NULL};
 119
 120 snprintf(buf, sizeof(buf), "PATH=%s", getenv("PATH"));
 121 execve("regression-test.py", argv, envp);

Setting gdb to follow that
(gdb) set follow-fork-mode child
(gdb) set detach-on-fork off
(gdb) run --automated-tests

And yeah it now is confirmed that the regression test always only "ran" the test (which if breaking on execution would flag a bug) but it never got to compare the actual output in the former modes.

So my upload will fix that to be on the same level as it was before - but we need upstream and some understanding if we can make it actually work (create valid ppm files with output that matches).

description: updated

There wasn't a spice project - but since spice-* is under one upstream umbrella spice-vdi is just as fine - important is that the gitlab tracker works.

Changed in spice (Ubuntu):
status: New → Triaged
Changed in spice (Ubuntu):
importance: Undecided → High
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package spice - 0.14.2-0ubuntu2

---------------
spice (0.14.2-0ubuntu2) eoan; urgency=medium

  * Fixup autpkgtest (LP: #1834286)
    These changes will make the test able to run again, but not output mismatch
    errors (this matches the behavior before 0.14.2). Upstream discussion
    started on how to resolve that as a next step, more details at the LP bug.
    - d/t/automated-tests: spice-common moved into dir subprojects
    - d/t/automated-tests: option --enable-automated-tests now is always on"
    - d/t/automated-tests, d/t/control: make tests more debuggable by allowing
      stderr
    - d/t/control: install new test dependency python-pil
    - d/t/base_test.ppm, d/t/regression-test.py: provide test resources from
      upstream git not part of the released tarball anymore
    - d/source/include-binaries: allow binary base_test.ppm in package

 -- Christian Ehrhardt <email address hidden> Tue, 25 Jun 2019 12:59:01 +0200

Changed in spice (Ubuntu):
status: Triaged → Fix Released
Changed in spice-vdi:
status: Unknown → New
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.