CI: OVB jobs fail because can't install XStatic from PyPI mirror on rh1 cloud

[Sagi (Sergey) Shnaidman]

All OVB jobs fail when trying to install XStatic-jQuery from http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi/packages (pypi mirror on rh1 cloud), while trying to install ara.

http://logs.openstack.org/60/508660/3/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq-newton/c061075/job-output.txt.gz#_2017-10-01_03_19_18_457466

HTTP error 404 while getting http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi/packages/f4/f3/b806d039ec4cd3ff601d7af49a18cb70697171c7b93030d4f1ffb881c174/XStatic-jQuery-1.10.2.1.tar.gz#md5=7a29efeab6961ef00ea2272c923f4503 (from http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi/simple/xstatic-jquery/)

http://logs.openstack.org/60/508660/3/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/7084b8f/job-output.txt.gz#_2017-10-01_03_19_49_801303

http://logs.openstack.org/60/508660/3/check-tripleo/legacy-tripleo-ci-centos-7-ovb-containers-oooq/a305639/job-output.txt.gz#_2017-10-01_03_19_46_984751

[yatin (yatinkarel)]

JFI: Local reproduction:-
virtualenv /tmp/ansible
/tmp/ansible/bin/pip install -i http://mirror.regionone.tripleo-test-cloud-rh1.openstack.org/pypi/simple/ --trusted-host mirror.regionone.tripleo-test-cloud-rh1.openstack.org ara

Tried with other cloud and it's working: /tmp/ansible/bin/pip install -i http://mirror.gra1.ovh.openstack.org/pypi/simple/ --trusted-host mirror.gra1.ovh.openstack.org ara

[Paul Belanger (pabelanger)]

Here was the fix:

  fs flushvolume /afs/openstack.org/mirror/pypi/

for some reason we needed to force an update of the cached data:

  http://docs.openafs.org/AdminGuide/HDRWQ412.html

[Ben Nemec (bnemec)]

Is that something we can do? And if not, would rebooting the mirror instance accomplish the same thing?

[Paul Belanger (pabelanger)]

Is there still an issue? The fix was to flush the AFS volume on the mirror.

[Ben Nemec (bnemec)]

No, I'm just wondering if this happens again if it's something we can resolve or if we need an infra admin.

[Ben Nemec (bnemec)]

I can confirm that the missing package is now found so we can close this.

[Sagi (Sergey) Shnaidman (sshnaidm)]

I'm not sure it's the same problem, but now ara fails to install in OVB because doesn't find libcffi:

http://logs.openstack.org/26/508226/1/check/legacy-tripleo-ci-centos-7-containers-multinode-upgrades/9642d03/job-output.txt.gz#_2017-10-04_08_17_31_722911

2017-10-04 08:17:31.722839 | primary | gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-x86_64-2.7/c/_cffi_backend.o
2017-10-04 08:17:31.722885 | primary | c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
2017-10-04 08:17:31.722911 | primary | #include <ffi.h>
2017-10-04 08:17:31.722936 | primary | ^
2017-10-04 08:17:31.722964 | primary | compilation terminated.
2017-10-04 08:17:31.723005 | primary | error: command 'gcc' failed with exit status 1