[infra] Tests fail on CI due to broken Internet connectivity: 'Cannot initiate the connection to archive.ubuntu.com'

[Artem Panchenko]

Sometimes system tests fail on CI because image assembling (for provisioning) or environment deployment returns error:

Cannot initiate the connection to archive.ubuntu.com:80 (91.189.92.200). - connect (101: Network is unreachable) [IP: 91.189.92.200 80]

Full trace: http://paste.openstack.org/show/478423/

In most cases deployment scripts perform re-tries, but nodes still can't access remote repositories. So the issue is more than just short network glitches.

[Igor Shishkin (teran)]

@Artem, I need more details.
Exact location this test was run, time.
Was it once or reproduced several time?

[Artem Panchenko (apanchenko-8)]

@Igor,

here is failed job buid: https://product-ci.infra.mirantis.net/job/8.0.ubuntu.smoke_neutron/176/

AFAIK It failed BVT only once, but on reverted environment `apt-get update` also returned error, so I believe mirrors are still broken.

Raising priority back to 'critical', because it causes failure of BVT.

[Artem Panchenko (apanchenko-8)]

please ignore my previous comment, it was targeted to bug #1518368

[Artem Panchenko (apanchenko-8)]

@Igor,

here is affected test job: https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.rollback_one_controller/25/testReport/(root)/ceph_ha_one_controller_compact/ceph_ha_one_controller_compact/

I've seen only one such failure on CI.

[Igor Shishkin (teran)]

Looks like the issue happened on master node/in the admin network.
Since network is unreachable ICMP message came from master node but our HN.
At the same time all the packages requested by two other nodes was successfully installed and, note, DNS resolve for archive.ubuntu.com passed successfully.

[Artem Panchenko (apanchenko-8)]

Sorry, I provided a wrong link above, original description contains traceback from this test/job:

https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.upgrade_one_controller_classic/26/testReport/%28root%29/ceph_ha_one_controller_compact/ceph_ha_one_controller_compact/

Anyway, I agree with you that domain name was resolved properly, so Internet connection worked fine (at least partially). However, I don't understand your statement 'ICMP message came from master node'. Yes, it was delivered to OS controller via default gw (master node), but we can't say which node sent that message, am I wrong?
From my point of view, most probably it was a network routing issue at DC's ISP or on the path to Ubuntu repo. This 'medium' priority bug was filed in order to improve or at least discuss improvement of our CI. It's obvious that unfortunately we can't completely avoid such sporadic network issues.

CI-team, maybe we can use our local mirrors of Ubuntu repositories in *all* SWARM jobs? Some of jobs are already using them, but https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.upgrade_one_controller_classic/ job (and some others actually) use default archive.ubuntu.com mirror.

[Aleksandra Fedorova (bookwar)]

As i understand this issue is happenning in upgrade/rollback jobs only. So path to ubuntu mirror should be fixed in upgrades template.

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Anton Tcitlionok <email address hidden>
Review: https://review.fuel-infra.org/14245

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14245
Submitter: Aleksandra Fedorova <email address hidden>
Branch: master

Commit: a3dcef51c07b9d1740dcdfdd9429a39f85f762db
Author: Anton Tcitlionok <email address hidden>
Date: Tue Dec 8 07:16:02 2015

Added logic for Ubuntu mirror configuration

Change-Id: I2ef4df84999ad27599c1ba879f6e6da20cd09f1e
Related-Bug: #1514830