s390x new ICEs due to not enough memory with parallel builds

Bug #1569750 reported by Timo Jyrinki on 2016-04-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
High
Martin Pitt
autopkgtest (Ubuntu)
Undecided
Unassigned
Dimitri John Ledkov (xnox) wrote :

Not ICE per-se, as no pre-processed source code is generated, nor e.g. it is retried. Just compiler is getting killed, probably due to out of memory as doko points out. Not reproducible in schroot on a zvm.

Martin Pitt (pitti) wrote :

Apparently this happens because the containers see 8 CPUs (and thus set DEB_BUILD_OPTIONS=parallel=8) but only have 1.8 GB of RAM, which blows up with memory-hungry C++ code.

affects: gcc-5 (Ubuntu) → auto-package-testing
Changed in auto-package-testing:
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) wrote :

Try to change LXC limits to only make one or two CPUs visible in the container, otherwise tell adt-run to use a smaller parallel= value.

Martin Pitt (pitti) on 2016-04-13
Changed in auto-package-testing:
importance: Undecided → High
Martin Pitt (pitti) on 2016-04-13
Changed in autopkgtest (Ubuntu):
status: New → In Progress
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) wrote :

Added new adt-run option in http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=6486fc5862

Now going to work on the glue to use this in the LXC runners.

Changed in autopkgtest (Ubuntu):
assignee: Martin Pitt (pitti) → nobody
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

I changed the worker to accept a new "extra_args" option in the config (https://git.launchpad.net/~ubuntu-release/+git/autopkgtest-cloud/commit/?id=86a510a), added "extra_args = --build-parallel=2" to the s390x config, and rolled it out.

A re-run now correctly shows "make -j2", but it still fails the same way:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/r/rocs/20160413_144415@/log.gz

Changed in auto-package-testing:
status: In Progress → Fix Released
Martin Pitt (pitti) wrote :

I reproduced the ICE manually in a container with "apt-get build-dep rocs" and "DEB_BUILD_OPTIONS=parallel=2 dpkg-buildpackage -us -uc -b". rocs still ICEs with that, but works with parallel=1.

So I suppose if g++ is that memory hungry on s390x, I'll have to disable parallelism entirely on s390x. I did that now, retried rocs, and it passes now:

  https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/r/rocs/20160413_151809@/log.gz

summary: - s390x new ICEs
+ s390x new ICEs due to not enough memory
summary: - s390x new ICEs due to not enough memory
+ s390x new ICEs due to not enough memory with parallel builds
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package autopkgtest - 3.20.4

---------------
autopkgtest (3.20.4) unstable; urgency=medium

  * setup-commands/setup-testbed: Fix kernel header installation for
    precise/armhf, which does not yet have linux-headers-generic.
  * Use "nproc" to determine the number of processors available, which is both
    more correct in a cgroup world, and also simpler.
  * Add new --build-parallel=N option to explicitly set the "parallel=N"
    $DEB_BUILD_OPTION for building packages, to override the default of
    "number of available CPUs". This is mostly useful in containers where you
    can restrict the available RAM, but not restrict the number of CPUs.
    (LP: #1569750)
  * setup-commands/setup-testbed: ubuntu-snappy got renamed to snapd, adjust
    package name.
  * adt-build-lxd: Set "distribution", "release", and "architecture"
    properties of generated images.
  * adt-build-lxd: Clean up all old images of the same distro/release/arch,
    not just the previous one. Also fix cleanup to work in non-English
    locales.
  * setup-commands/setup-testbed: Call apt-get purge only once with the list
    of all packages, instead of once per package. This is much faster.
  * SchrootClickRunner tests: Chown click dir in /opt after creating the user.
  * Bump Standards-Version to 3.9.8 (no changes necessary).
  * Update Vcs-* URLs.

 -- Martin Pitt <email address hidden> Mon, 18 Apr 2016 11:19:45 +0200

Changed in autopkgtest (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers