autopkgtest fails in impish (0.22)

[Christian Ehrhardt ]

There are (at least) two issues why the tests fail in Impish right now.

It blocks xtensor itself, but also src:xtensor-blas.

The test fails on all architectures, but for different reasons.
The last good test was more than two years ago.

s390x / arm64 / i386 fail on bad-package.
  - xtensor-dev became a transitional and is available on all arch.
  - But the actual dependency libxtensor-dev only exists on a few
    libxtensor-dev | 0.22.0-5 | impish-proposed | amd64, armhf, ppc64el, riscv64
  - That explains the badpkg issue, and I guess we can just reset it
    instead of spending too much time fixing the package.
  - TODO test reset on those architectures

- Furthermore amd64 / ppc64 tests fail while armhf works.
    https://autopkgtest.ubuntu.com/packages/x/xtensor/impish/amd64
  - In debici this passes on amd64/arm64/armel/armhf/ppc64 but fails i386/s390x.
    https://ci.debian.net/packages/x/xtensor/
  - It appears to be a build error at first, but it really is an OOM (log below).
  - The problem that this isn't just "big packages and being ok" I tried with 3G/4G/8G
    memory. 4G still fails, 8G works so it is not an infinite-memory bug.
  - TODO mark as big package once 8G is confirmed

The test also has some control about its concurrency, but I tried to reduce it to -j 1 and it still fails with an OOM. But then it works with 3G which is an improvement.
TODO: submit to Debian to reduce the nproc in this test case.

[ 326.699865] Swap cache stats: add 0, delete 0, find 0/0
[ 326.699866] Free swap = 0kB
[ 326.699867] Total swap = 0kB
[ 326.699868] 1048443 pages RAM
[ 326.699869] 0 pages HighMem/MovableOnly
[ 326.699870] 43233 pages reserved
[ 326.699870] 0 pages hwpoisoned
[ 326.699871] Tasks state (memory values in pages):
[ 326.699872] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 326.699877] [ 280] 0 280 16183 668 110592 0 -250 systemd-journal
[ 326.699881] [ 305] 0 305 5604 875 61440 0 -1000 systemd-udevd
[ 326.699886] [ 317] 100 317 4003 490 77824 0 0 systemd-network
[ 326.699889] [ 398] 0 398 53869 4659 90112 0 -1000 multipathd
[ 326.699891] [ 426] 101 426 6025 1470 86016 0 0 systemd-resolve
[ 326.699894] [ 427] 102 427 22107 708 73728 0 0 systemd-timesyn
[ 326.699896] [ 432] 0 432 2079 1126 53248 0 0 haveged
[ 326.699898] [ 435] 0 435 657 375 40960 0 0 sh
[ 326.699900] [ 437] 0 437 1721 527 53248 0 0 cron
[ 326.699902] [ 439] 103 439 2125 941 53248 0 -900 dbus-daemon
[ 326.699906] [ 444] 0 444 20730 685 57344 0 0 irqbalance
[ 326.699908] [ 445] 0 445 7821 2956 102400 0 0 networkd-dispat
[ 326.699910] [ 446] 0 446 976 245 49152 0 0 rngd
[ 326.699912] [ 447] 104 447 55304 972 77824 0 0 rsyslogd
[ 326.699914] [ 448] 0 448 3450 608 61440 0 0 systemd-logind
[ 326.699915] [ 449] 0 449 98458 1307 118784 0 0 udisksd
[ 326.699917] [ 485] 0 485 3324 816 65536 0 -1000 sshd
[ 326.699919] [ 486] 0 486 1406 442 53248 0 0 agetty
[ 326.699921] [ 487] 0 487 1406 441 53248 0 0 agetty
[ 326.699923] [ 489] 0 489 1498 391 53248 0 0 agetty
[ 326.699924] [ 494] 0 494 58746 916 90112 0 0 polkitd
[ 326.699926] [ 644] 0 644 73985 1378 167936 0 0 packagekitd
[ 326.699928] [ 1668] 0 1668 657 33 40960 0 0 sh
[ 326.699931] [ 1669] 0 1669 2119 533 57344 0 0 su
[ 326.699932] [ 1671] 1000 1671 3839 957 73728 0 0 systemd
[ 326.699934] [ 1672] 1000 1672 25024 748 94208 0 0 (sd-pam)
[ 326.699936] [ 1677] 1000 1677 2280 492 61440 0 0 bash
[ 326.699938] [ 1684] 1000 1684 2009 516 57344 0 0 make
[ 326.699940] [ 1685] 1000 1685 2280 235 53248 0 0 bash
[ 326.699942] [ 1686] 1000 1686 2280 230 53248 0 0 bash
[ 326.699943] [ 1687] 1000 1687 1850 77 49152 0 0 tee
[ 326.699945] [ 1688] 1000 1688 1850 84 49152 0 0 tee
[ 326.699947] [ 1887] 1000 1887 2009 521 45056 0 0 make
[ 326.699949] [ 1890] 1000 1890 2009 543 49152 0 0 make
[ 326.699951] [ 1934] 1000 1934 2070 560 57344 0 0 make
[ 326.699953] [ 2179] 1000 2179 1701 23 45056 0 0 g++
[ 326.699954] [ 2180] 1000 2180 379282 366807 3063808 0 0 cc1plus
[ 326.699956] [ 2184] 1000 2184 1701 22 49152 0 0 g++
[ 326.699958] [ 2185] 1000 2185 437147 423981 3530752 0 0 cc1plus
[ 326.699960] [ 2199] 1000 2199 1701 22 57344 0 0 g++
[ 326.699962] [ 2200] 1000 2200 109270 97183 901120 0 0 cc1plus
[ 326.699964] [ 2204] 1000 2204 1701 22 49152 0 0 g++
[ 326.699965] [ 2205] 1000 2205 68071 56459 573440 0 0 cc1plus
[ 326.699967] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-c2.scope,task=cc1plus,pid=2185,uid=1000
[ 326.699988] Out of memory: Killed process 2185 (cc1plus) total-vm:1748588kB, anon-rss:1695328kB, file-rss:596kB, shmem-rss:0kB, UID:1000 pgtables:3448kB oom_score_adj:0

[Christian Ehrhardt  (paelzer)]

The three TODOs that were identified are submitted.

Test reset on architectures that miss libxtensor-dev:
=> https://code.launchpad.net/~paelzer/britney/+git/hints-ubuntu/+merge/405894

Mark as big_package to provide the test more memory:
=> https://code.launchpad.net/~paelzer/autopkgtest-cloud/+git/autopkgtest-package-configs/+merge/405895

Submit to Debian to reduce the nproc in this test case:
=> https://salsa.debian.org/science-team/xtensor/-/merge_requests/1

Once at least the first two completed - and if big packages is only 4G (I was unable to find details today) then also the MR to Debian - are completed then these tests should work fine again (at least if the result of the local repro will work out the same in autopkgtest infrastructure).

[Christian Ehrhardt  (paelzer)]

With all hints in place I got two passes now:
https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/amd64/x/xtensor/20210720_065346_146d0@/log.gz
https://autopkgtest.ubuntu.com/results/autopkgtest-impish/impish/ppc64el/x/xtensor/20210720_065542_50c0e@/log.gz

Rechecking later if that also fully unblocked the migration to impish-release

[Christian Ehrhardt  (paelzer)]

It completed 7 minutes ago:
2021-07-20 11:53:50 CEST Published Impish release universe misc 0.22.0-5

Thereby this bug can be marked fixed.