webbrowser-app’s unit tests crash in CI jobs: access to /dev/shm is denied

[Olivier Tilloy]

The CI job that runs autopilot tests for webbrowser-app on otto (desktop) has been consistently failing since the switch to vivid.
The error seen in the log is the following:

/var/local/autopilot/autopilot.log: [1127/102329:ERROR:shared_memory_posix.cc(231)] Creating shared memory in /dev/shm/.org.chromium.Chromium.UAVbQt failed: Permission denied
/var/local/autopilot/autopilot.log: [1127/102329:ERROR:shared_memory_posix.cc(234)] Unable to access(W_OK|X_OK) /dev/shm: Permission denied
/var/local/autopilot/autopilot.log: [1127/102329:FATAL:shared_memory_posix.cc(236)] This is frequently caused by incorrect permissions on /dev/shm. Try 'sudo chmod 1777 /dev/shm' to fix.

At a first glance it looks to me like the problem is in the otto setup itself.

[Olivier Tilloy (osomon)]

Interestingly, I’m seeing the same error when running webbrowser-app’s unit tests inside a vivid chroot.
The permissions for /dev/shm there are "drwxr-xr-x".

[Olivier Tilloy (osomon)]

Oxide, which is based on chromium, needs write-access to /dev/shm to function properly. So we need to ensure /dev/shm is writeable in our test environment.

[Olivier Tilloy (osomon)]

Additionally, i386 and and amd64 builds now also fail in CI, because of the same error (interestingly they pass on armhf, I guess because they are being run on real hardware as opposed to chroots for the other architectures). See e.g. https://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-amd64-ci/86/console.

[Francis Ginther (fginther)]

Discussed with Olivier over IRC that the otto issue is a limitation of the lxc containment in use and further support of otto is being deprecated in favor of a future solution that will run desktop tests natively on the hardware. Olivier also supports just disabling these tests (which was guidance we received from bfiller at the UE sprint in Washington DC) as they have just become noise and the more useful tests are the ones being run on touch devices.

I will need to investigate the /dev/shm issue for the vivid builds. My theory is that this used to work but was disabled as it presents a potential containment hole between the chroot and the host. But that's just a theory. Given how we setup the pbuilder chroots, we may be able to do something about this. My only concern is that LP may then fail to build due to this problem. But currently Olivier can confirm that LP still builds.

[Olivier Tilloy (osomon)]

It’s likely that LP virtual builders for x86 are on trusty (or an older ubuntu release anyway), this /dev/shm issue started appearing in vivid, which would explain why we’re not seeing it in PPA builds.

Of course when LP builders are upgraded to vivid(+n), builds will start failing. If we can devise an acceptable solution for the vivid chroots used by CI, then we should be able to apply it to LP as well.

[Olivier Tilloy (osomon)]

In a utopic chroot, /dev/shm is a symlink to /run/shm:

lrwxrwxrwx 1 root root 8 Apr 29 2014 /dev/shm -> /run/shm
drwxrwxrwt 2 root root 40 Dec 5 11:51 /run/shm

Whereas in vivid it is a directory, with no write permissions for group and other:

drwxr-xr-x 2 root root 4096 Nov 7 10:21 /dev/shm

[Olivier Tilloy (osomon)]

For reference:

<pitti> oSoMoN: that looks ok; is /dev/shm a symlink?
<oSoMoN> pitti, no, /dev/shm is not a symlink, it’s a plain directory
 (drwxr-xr-x)
<ogra_> aha
 and not 1777
<oSoMoN> yup
 but I have no idea why it doesn’t have the right permissions
<oSoMoN> (which is my original question)
<pitti> ogra_: ah, it should be a symlink
<pitti> err, oSoMoN
<pitti> well, it really shouldn't be, but with Debian's layout it should
<oSoMoN> pitti, do you know what is responsible for creating it in the first place, when creating a chroot?
<pitti> oSoMoN: it depends on how you build it, but I figure somewhere between initscript's postinst and debootstrap; mk-sbuild and friends might also make some adjustments there, and sbuild has its own fstabs

[Francis Ginther (fginther)]

I dug some more on the /dev/shm not a symlink and filed:
https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/1399772

Also setting otto to won't fix as it's being deprecated.

[Olivier Tilloy (osomon)]

Now that bug #1399772 has been fixed, the vivid chroots used by CI jobs need to be rebuilt to ensure that /dev/shm is created with the correct permissions.

[Olivier Tilloy (osomon)]

It seems the amd64 chroot was rebuilt, unit tests now pass. The i386 chroot still suffers from the same problem though.

[Olivier Tilloy (osomon)]

All vivid chroots have been rebuilt now, closing the issue as invalid. Thanks Francis!