test gets stuck with 'No space left on device'

[Balint Reczey]

Test is run, shows up on http://autopkgtest.ubuntu.com/running , but the results never make it to http://autopkgtest.ubuntu.com/package/... without admin intervention.

Observed with firefox:
...
autopkgtest [00:49:41]: @@@@@@@@@@@@@@@@@@@@ summary
firefox-version PASS
html5test PASS
search-engines PASS
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 848, in <module>
    main()
  File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 843, in main
    create_testinfo(vserver_args)
  File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 223, in create_testinfo
    json.dump(info, f, indent=2)
OSError: [Errno 28] No space left on device

I have no link to the full log, because of the stuckage. :-)

Data I could gather:

firefox
Release: bionic
Architecture: armhf
Submit-time: 2020-12-10 21:53:47
Triggers: ['glibc/2.27-3ubuntu1.4']
Requester: rbalint
Running for: 11h 3m 32s

[Iain Lane (laney)]

We'll mitigate this on the "new" environment, by asking for instances
with way more disk available, but autopkgtest should have tmpfailed here
of course. That's a bug in autopkgtest itself IMO. Even the timeout
doesn't fire when this happens AFAIK.

The problem on the autopkgetst-cloud side is the runner itself runs on a
machine which is shared between all running tests. In the worst case
this instance needs enough space to store the source, potentially the
binaries, the logs and any artifacts from all currently running tests.
If we have (e.g.) a lot of Libreoffices running, which are a few gigs
each, then we can fill up our disk.

Related old Trello card:

  https://trello.com/c/1UFK93tT/25-handle-enospc

On Fri, Dec 11, 2020 at 11:05:07AM -0000, Balint Reczey wrote:
> Public bug reported:
>
> Test is run, shows up on http://autopkgtest.ubuntu.com/running , but the
> results never make it to http://autopkgtest.ubuntu.com/package/...
> without admin intervention.
>
> Observed with firefox:
> ...
> autopkgtest [00:49:41]: @@@@@@@@@@@@@@@@@@@@ summary
> firefox-version PASS
> html5test PASS
> search-engines PASS
> OSError: [Errno 28] No space left on device
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 848, in <module>
> main()
> File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 843, in main
> create_testinfo(vserver_args)
> File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 223, in create_testinfo
> json.dump(info, f, indent=2)
> OSError: [Errno 28] No space left on device
>
> I have no link to the full log, because of the stuckage. :-)
>
> Data I could gather:
>
> firefox
> Release: bionic
> Architecture: armhf
> Submit-time: 2020-12-10 21:53:47
> Triggers: ['glibc/2.27-3ubuntu1.4']
> Requester: rbalint
> Running for: 11h 3m 32s
>
> ** Affects: auto-package-testing
> Importance: Undecided
> Status: New
>
> ** Affects: autopkgtest (Debian)
> Importance: Unknown
> Status: Unknown
>
> ** Bug watch added: Debian Bug tracker #927146
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=927146
>
> ** Also affects: autopkgtest (Debian) via
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=927146
> Importance: Unknown
> Status: Unknown
>
> --
> You received this bug notification because you are subscribed to Auto
> Package Testing.
> https://bugs.launchpad.net/bugs/1907803
>
> Title:
> test gets stuck with 'No space left on device'
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/auto-package-testing/+bug/1907803/+subscriptions

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

[Balint Reczey (rbalint)]

Autopkgtest fix:
https://salsa.debian.org/rbalint/autopkgtest/-/pipelines/206254