Autopkgtest fails on s390x due to long PATH in test config

[Balint Reczey]

...
2017-03-29T08:40:18.627231Z 0 [Warning] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2017-03-29T08:40:18.627257Z 0 [ERROR] The socket file path is too long (> 107): /data/adttmp/autopkgtest-virt-lxc.shared.y4ui3net/downtmp/build.R3s/libdbd-mysql-perl-4.041/t/testdb/mysql.sock
2017-03-29T08:40:18.627267Z 0 [ERROR] Aborting

2017-03-29T08:40:18.627278Z 0 [Note] Binlog end
2017-03-29T08:40:18.627315Z 0 [Note] Shutting down plugin 'ngram'
2017-03-29T08:40:18.627328Z 0 [Note] Shutting down plugin 'partition'
2017-03-29T08:40:18.627337Z 0 [Note] Shutting down plugin 'BLACKHOLE'
...

[Balint Reczey (rbalint)]

Tried the approach with creating temp dir with mktemp -d but both smoke-setup and smoke-cleanup have to use the same temp dir this is complicated.
The better fix will be shortening path components set by autopkgtest.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "libdbd-mysql-perl_4.041-1ubuntu1.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Dimitri John Ledkov (xnox)]

On 7 April 2017 at 13:58, Balint Reczey <email address hidden> wrote:
> Tried the approach with creating temp dir with mktemp -d but both smoke-setup and smoke-cleanup have to use the same temp dir this is complicated.
> The better fix will be shortening path components set by autopkgtest.
>

To achieve shorter path, please check autopkgtest source code. Indeed
this is not the only test that has been failing like this.

--
Regards,

Dimitri.

[Balint Reczey (rbalint)]

Reassigning to fix it in autopkgtest only

[Dimitri John Ledkov (xnox)]

On 11 April 2017 at 14:58, Balint Reczey <email address hidden> wrote:
> ** Patch added: "autopkgtest_4.3ubuntu2.patch"
> https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1680577/+attachment/4860202/+files/autopkgtest_4.3ubuntu2.patch
>

Hm, no. One should not use the old adt name, and the saving here is
fairly minimal.

One should look to decrease the paths and subdirs used in that temp
dir and try to shorten that too, to not like encode the full package
name, version, testcase name, etc. and just have "/src/" subdir there.

--
Regards,

Dimitri.

[Balint Reczey (rbalint)]

Autopkgtest needs to be run in extracted source thus the following path part is on the package:
"libdbd-mysql-perl-4.041/t/testdb/mysql.sock"

This comes from the system configuration:
"/data/adttmp/"

This is what autopkgtest generates (55 chars from the max 107):
"autopkgtest-virt-lxc.shared.y4ui3net/downtmp/build.R3s"

Where do you see better opportunities for saving characters and which abbreviation would be acceptable?

[Balint Reczey (rbalint)]

I'm adding a patch to use "src" for the package dir, but i think cutting from autopkgtest-virt-lxc.shared would make sense, too.

[Balint Reczey (rbalint)]

I have proposed the patch cutting the build directory name to "src" in the Debian bug.
Waiting for comments from the maintainter.

(Attaching here, too)

[Mathieu Trudel-Lapierre (cyphermox)]

How would that impact running multiple different instances of autopkgtest locally given keeping the data after a run? Not having the name of the package for which a run seems to make things more confusing: you'd need to track which temporary directory was used for which run, or go dig in the debian/control file.

Seems like the right location here for gains really is to shorten the test harness' directory; from say 'autopkgtest-virt-lxc' to just 'lxc'? The same logic appears to seemlessly apply to other harnesses; like qemu. We have the potential to gain as much as 17 characters here, without sacrificing in clarity of what things where used for in /tmp.

Also, have you discussed this with upstream / directly with pitti?

[Balint Reczey (rbalint)]

> How would that impact running multiple different instances of
> autopkgtest locally given keeping the data after a run? Not having the
> name of the package for which a run seems to make things more confusing:
> you'd need to track which temporary directory was used for which run, or
> go dig in the debian/control file.

Yes, this would be a bit harder to tell which dir holds a given package,
but "grep Source: ...path.../*/debian/control" is not that complicated.

> Seems like the right location here for gains really is to shorten the
> test harness' directory; from say 'autopkgtest-virt-lxc' to just 'lxc'?
> The same logic appears to seemlessly apply to other harnesses; like
> qemu. We have the potential to gain as much as 17 characters here,
> without sacrificing in clarity of what things where used for in /tmp.

The maximal gain is much more, the package names and versions can get really long:

 udd=> select max(length(source)) , max(length(source_version)) from packages;
  max | max
 -----+-----
   56 | 69
 (1 row)

I think it would be reasonable to cut from the test harness directory, too, but
xnox liked this approach better.

> Also, have you discussed this with upstream / directly with pitti?
>

I initiated the discussion in the Debian bug, but got little response so far.
Note that pitti is looking for help with the package:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=846328

[Martin Pitt (pitti)]

> This is what autopkgtest generates (55 chars from the max 107):
> "autopkgtest-virt-lxc.shared.y4ui3net/downtmp/build.R3s"
> Where do you see better opportunities for saving characters and which abbreviation would be acceptable?

Both places can/should be shortened. The simple part is the shared temp dir, I committed

  https://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=7e55125bb9fd9

This should already be sufficient for this particular mysql case.

I also agree to your patch, but it breaks (at least) the ChrootRunner tests as the mock apt-get source needs to be adjusted. I'm working on this. Thanks!

[Martin Pitt (pitti)]

This also breaks the schroot tests, which looks more serious:

$ tests/autopkgtest SchrootRunner.test_apt_source
[...]
Download complete and in download only mode
Get:1 http://httpredir.debian.org/debian sid/main gdk-pixbuf 2.36.5-2 (dsc) [2781 B]
Get:2 http://httpredir.debian.org/debian sid/main gdk-pixbuf 2.36.5-2 (tar) [5166 kB]
Get:3 http://httpredir.debian.org/debian sid/main gdk-pixbuf 2.36.5-2 (diff) [14.0 kB]
+ dpkg-source -x gdk-pixbuf_.dsc src
dpkg-source: error: cannot read gdk-pixbuf_.dsc: No such file or directory

[Simon Quigley (tsimonq2)]

Unsubscribing ~ubuntu-sponsors as there doesn't look like there's anything left to be sponsored. If this was done in error, feel free to resubscribe ~ubuntu-sponsors.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 5.0

---------------
autopkgtest (5.0) unstable; urgency=medium

  [ Martin Pitt ]
  * Drop obsolete adt-* CLI
    autopkgtest 4.0 with its "autopkgtest" program has been around for a
    year, Debian's and Ubuntu's CI moved to that, and the Debian 9.0 release
    is behind us. Drop "adt-run" and the corresponding adt-virt-* and
    adt-build-* command line interfaces now.
    Add Breaks: for debci for versions that still relied on adt-*.
  * lxc, qemu, ssh: Shorten path of shared directory.
    This helps to reduce socket path lengths in tests.
    (See Debian #860554, LP: #1680577)
  * Drop obsolete ubuntu-touch-session setup script.
    This hasn't been maintained/tested for ages, and with the demise of
    Ubuntu Touch this is now completely obsolete. (Closes: #835836)
  * Avoid bare "except:" clauses
  * tests/autopkgtest_args: pyflakes/pycodestyle fixes
  * tests: Don't run LXC tests with --ephemeral.
    It's not the default and ephemeral containers tend to fail to start too
    often.
  * tests/testpkg: Update Standards-Version and debhelper level.
    Priority "extra" is deprecated, so move to "optional".
  * README.package-tests.rst: Clarify role of control file, add simple
    examples (Closes: #870654)
  * README.package-tests.rst: Update Testsuite: documentation.
    All supported Debian releases now have a new enough dpkg, and Ubuntu
    14.04 will not get new tests any more. (Closes: #876008)
  * Run apt-ftparchive for local debs on testbed instead of host.
    This is slightly more correct as it will produce the format that the
    testbed's apt expects (although that doesn't matter much). It also
    allows autopkgtest to be run on non-Debian hosts.
    As apt-utils is not essential, install it if necessary.
  * README.package-tests.rst: Clarify Testsuite: values
    `autopkgtest` is not any more the only defined value for Testsuite:, as
    the paragraph below already explains.
  * setup-commands/setup-testbed: Look at /etc/os-release for determining
    distribution name. Stop hardcoding "debian|ubuntu|kali" and get this
    information from os-release's ID field. (Closes: #868678)
  * debian/rules: Drop obsolete pysupport override.
    Not necessary any more since dh compat level 9.
  * Bump Standards-Version to 4.1.0. (No changes necessary.)

  [ Stéphane Graber ]
  * debian/tests/lxd: Update to support LXD network/storage API

  [ Steve Langasek ]
  * Fix release name parsing with arch-qualified sources.list
    (Closes: #870393)

  [ Antonio Terceiro ]
  * autopkgtest-build-lxc: Correctly clean cache for Debian containers

 -- Martin Pitt <email address hidden> Sun, 17 Sep 2017 16:50:21 +0200