Rackspace package mirror periodically inconsistent causing gate build failures

[Kieran Spear]

2013-11-14 00:09:57.103 | 2013-11-14 00:09:57 + sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get --option Dpkg::Options::=--force-confold --assume-yes install make fakeroot dkms openvswitch-switch openvswitch-datapath-dkms linux-headers-3.2.0-56-virtual
2013-11-14 00:09:57.104 | 2013-11-14 00:09:57 Reading package lists...
2013-11-14 00:09:57.104 | 2013-11-14 00:09:57 Building dependency tree...
2013-11-14 00:09:57.105 | 2013-11-14 00:09:57 Reading state information...
2013-11-14 00:09:57.105 | 2013-11-14 00:09:57 fakeroot is already the newest version.
2013-11-14 00:09:57.106 | 2013-11-14 00:09:57 fakeroot set to manually installed.
2013-11-14 00:09:57.107 | 2013-11-14 00:09:57 linux-headers-3.2.0-56-virtual is already the newest version.
2013-11-14 00:09:57.112 | 2013-11-14 00:09:57 make is already the newest version.
2013-11-14 00:09:57.116 | 2013-11-14 00:09:57 make set to manually installed.
2013-11-14 00:09:57.121 | 2013-11-14 00:09:57 The following extra packages will be installed:
2013-11-14 00:09:57.146 | 2013-11-14 00:09:57 openvswitch-common
2013-11-14 00:09:57.147 | 2013-11-14 00:09:57 Suggested packages:
2013-11-14 00:09:57.147 | 2013-11-14 00:09:57 ethtool
2013-11-14 00:09:57.148 | 2013-11-14 00:09:57 The following NEW packages will be installed:
2013-11-14 00:09:57.149 | 2013-11-14 00:09:57 dkms openvswitch-common openvswitch-datapath-dkms openvswitch-switch
2013-11-14 00:09:57.149 | 2013-11-14 00:09:57 0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
2013-11-14 00:09:57.154 | 2013-11-14 00:09:57 Need to get 3,900 kB of archives.
2013-11-14 00:09:57.158 | 2013-11-14 00:09:57 After this operation, 17.0 MB of additional disk space will be used.
2013-11-14 00:09:57.163 | 2013-11-14 00:09:57 Get:1 http://mirror.rackspace.com/ubuntu/ precise-updates/main dkms all 2.2.0.3-1ubuntu3.1 [73.2 kB]
2013-11-14 00:09:57.167 | 2013-11-14 00:09:57 Get:2 http://mirror.rackspace.com/ubuntu/ precise-updates/universe openvswitch-common amd64 1.4.0-1ubuntu1.5 [547 kB]
2013-11-14 00:09:57.185 | 2013-11-14 00:09:57 Err http://mirror.rackspace.com/ubuntu/ precise-updates/universe openvswitch-datapath-dkms all 1.4.0-1ubuntu1.5
2013-11-14 00:09:57.187 | 2013-11-14 00:09:57 Could not connect to mirror.rackspace.com:80 (74.205.112.120). - connect (110: Connection timed out)
2013-11-14 00:10:00.230 | 2013-11-14 00:10:00 Err http://mirror.rackspace.com/ubuntu/ precise-updates/universe openvswitch-switch amd64 1.4.0-1ubuntu1.5
2013-11-14 00:10:00.269 | 2013-11-14 00:10:00 Unable to connect to mirror.rackspace.com:http:
2013-11-14 00:10:00.270 | 2013-11-14 00:10:00 Failed to fetch http://mirror.rackspace.com/ubuntu/pool/universe/o/openvswitch/openvswitch-datapath-dkms_1.4.0-1ubuntu1.5_all.deb Could not connect to mirror.rackspace.com:80 (74.205.112.120). - connect (110: Connection timed out)
2013-11-14 00:10:00.271 | 2013-11-14 00:10:00 Failed to fetch http://mirror.rackspace.com/ubuntu/pool/universe/o/openvswitch/openvswitch-switch_1.4.0-1ubuntu1.5_amd64.deb Unable to connect to mirror.rackspace.com:http:
2013-11-14 00:10:00.271 | 2013-11-14 00:10:00 Fetched 620 kB in 1min 21s (7,583 B/s)
2013-11-14 00:10:00.271 | 2013-11-14 00:10:00 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

This is probably expected from time to time, but is there a way to detect it and abort the build?

[Kieran Spear (kspear)]

http://logs.openstack.org/99/56299/1/check/check-tempest-devstack-vm-neutron/f060a45/console.html
https://review.openstack.org/#/c/56299/1

[Mike Perez (thingee)]

http://logs.openstack.org/06/57406/2/check/check-tempest-devstack-vm-neutron/590e7e0
https://review.openstack.org/#/c/57406/2

[Samuel Merritt (torgomatic)]

http://logs.openstack.org/50/57050/3/check/check-tempest-devstack-vm-full/9aa3c90

https://review.openstack.org/#/c/57050/3

[Davanum Srinivas (DIMS) (dims-v)]

Sigh. started again

http://logs.openstack.org/73/106073/1/check/check-grenade-dsvm/0d51036/logs/grenade.sh.txt.gz#_2014-07-10_15_10_25_034

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/106196

[OpenStack Infra (hudson-openstack)]

Change abandoned by Davanum Srinivas (dims) (<email address hidden>) on branch: master
Review: https://review.openstack.org/106196

[Jeremy Stanley (fungi)]

This is actually not a duplicate of 1286818. That bug is for problems with archive.ubuntu.com rather than mirror.rackspace.com. We may want to keep these bugs separated?

[Matt Riedemann (mriedem)]

We could keep these separate bugs but there is an elastic-recheck query against "E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?" for bug 1286818 so most people are probably going to recheck against that (since we don't have multi-line query support we can't say if it's rax or ubuntu in the logstash query). Maybe we could filter with the build_node filter in logstash?

[Jordan Pittier (jordan-pittier)]

I see a lot of this :

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiSFRUUCBjaGVjayBvZiBodHRwOi8vbWlycm9yLnJhY2tzcGFjZS5jb20vdWJ1bnR1L2Rpc3RzL3RydXN0eS9SZWxlYXNlLmdwZyAtIGF0dGVtcHQgIzdcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiMTQ0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI2ODY5ODk2MDI0fQ==

message:"HTTP check of http://mirror.rackspace.com/ubuntu/dists/trusty/Release.gpg - attempt #7"

It's not stable...