rm: cannot remove `/home/jenkins/.cache/image-create/pypi/mirror/': Permission denied

[James Polley]

CI is failing on all jobs with error messages like this:

2014-12-24 16:21:54.964 | + popd
2014-12-24 16:21:54.964 | /opt/stack/new/tripleo-ci
2014-12-24 16:21:54.964 | + export TRIPLEO_ROOT=/opt/stack/new/
2014-12-24 16:21:54.964 | + TRIPLEO_ROOT=/opt/stack/new/
2014-12-24 16:21:54.964 | + MIRROR_ROOT=/home/jenkins/.cache/image-create/pypi/mirror/
2014-12-24 16:21:54.965 | + rm -rf /home/jenkins/.cache/image-create/pypi/mirror/
2014-12-24 16:21:54.987 | rm: cannot remove `/home/jenkins/.cache/image-create/pypi/mirror/': Permission denied
2014-12-24 16:21:54.992 | ERROR: the main setup script run by this job failed - exit code: 1
2014-12-24 16:21:54.992 | please look at the relevant log files to determine the root cause

http://logs.openstack.org/43/141043/6/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/36c9771/console.html is the earliest example I can find.

[James Polley (tchaypo)]

https://jenkins03.openstack.org/job/check-tripleo-ironic-overcloud-precise-nonha/676/console shows:

2014-12-26 08:27:14.123 | + id
2014-12-26 08:27:14.133 | uid=2015(jenkins) gid=2015(jenkins) groups=2015(jenkins)
2014-12-26 08:27:14.150 | + find /home/jenkins/.cache -type d -ls
2014-12-26 08:27:14.150 | 415945 4 drwx------ 3 root root 4096 Dec 25 14:39 /home/jenkins/.cache
2014-12-26 08:27:14.150 | find: `/home/jenkins/.cache': Permission denied

So that explains why this is a problem, but I still can't figure out what changed to make this start happening.

[Adam Gandelman (gandelman-a)]

Looks like this is the first formal bug filed about it, but this was discovered last week after a new upstream release of pip. pip now creates and uses $HOME/.cache for cache stuff. DIB also uses $HOME/.cache. If a script calls 'sudo pip install' prior to a DIB run, DIB will fail to use the default cahce dir. This first turned up on some ironic gate jobs, and then again in trove's integration tests (both use 'sudo pip install ...' followed by a DIB build. The solution is to pass -H to sudo during pip installs to ensure its cache dir is at a non-conflicting /root/.cache/

see related patches:

https://review.openstack.org/#/c/143501/
https://review.openstack.org/#/c/143746/

[James Polley (tchaypo)]

This sounds plausible, but I can't find anywhere in tripleo-ci where we're running 'sudo pip', so there's nothing I can change here. http://logs.openstack.org/43/141043/6/check-tripleo/check-tripleo-ironic-overcloud-precise-nonha/36c9771/console.html doesn't show "sudo pip" either.

So although this seems plausible, I can't find any pip installs that need a -H

[James Polley (tchaypo)]

I've looked through tripleo-incubator, tripleo-image-elements, and diskimage-builder, but I can't find anything else matching "sudo.*pip"

[James Polley (tchaypo)]

My last comment was written with the in-head context of having just pushed https://review.openstack.org/#/c/144107/, which fixed all the instances of "sudo pip" i could find in openstack-infra/project-config

If I'd put that context into the comment, it would have made more sense.

In any case, https://review.openstack.org/#/c/144107/ has just landed. My understanding is that we need to wait for new nodepool images to be built (~20 hours from now, if nodepool can be nursed back to health beforehand), after which (if 144107 fixed the source of the problem) CI should start passing again.

[Adam Gandelman (gandelman-a)]

Tagging as affecting devstack as well, this is cropping up in other places where pip and DIB are used together.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/144216

[Adam Gandelman (gandelman-a)]

Nevermind, this one looks more specific to tripleo's usage (failing on rm'ing vs mkdir)

[James Polley (tchaypo)]

I believe this was fixed by both https://review.openstack.org/#/c/144184/ and https://review.openstack.org/#/c/144107/ - either change on its own should have resolved the issue (144107 required a nodepool image rebuild, 144184 should have been immediately effective).