Sporadic external connectivity failures in staging jobs

[Roman Podoliaka]

http://jenkins-product.srt.mirantis.net:8080/view/All/job/5.1.2.test_staging_mirror/202 failed with

AssertionError: Failed tests, fails: 1 should fail: 0 failed tests name: [{u'Check network connectivity from instance via floating IP (failure)': u'Time limit exceeded while waiting for public connectivity checking from VM to finish. Please refer to OpenStack logs for more details.'}]

Looks like we hit a temporarily connectivity issue:

http://paste.openstack.org/show/332206/

New instances work just fine.

[Vladimir Sharshov (vsharshov)]

Looks like have same nature as https://bugs.launchpad.net/fuel/+bug/1471812

[Vladimir Sharshov (vsharshov)]

Guys, i mark it as High because false alarm for CI is distracting. Feel free to set another importance priority.

[Igor Shishkin (teran)]

Roman, please ensure you have proper network settings on Fuel Master node and your Controllers compute.
For that purpose please attach output of the following command:

On Fuel Master node:
ip a
ping 8.8.8.8
arp -an

On other nodes:
ip a
ping $MASTERNODE_IP
arp -an

Marking as incomplete and assigning to author since there is no enough details.

[Roman Podoliaka (rpodolyaka)]

Igor, I reported this bug when our team was on the staging mirrors duty, so I don't have any environments for you.

My point here is that it's not a problem with settings on the master node, but rather *irregular* issues with external connectivity, which happen time to time on our Jenkins slaves.

The only reason I filed this bug is that we have this policy about attaching a bug entry to each Jenkins job failure. But as we see, there is interest for this as these connectivity issues continue to happen.

[Roman Podoliaka (rpodolyaka)]

I suggest we just mark the failed jobs in a special way, rather than continue to file new and new LP issues for the very same problem with external connectivity.

And on fuel-devops team side it's probably worth investigation why those connectivity issues happen rather often on the Jenkins slaves.

[Igor Shishkin (teran)]

moving to 8.0 since there is no real progress started and it doesn't affect 7.0 release

[Igor Shishkin (teran)]

Do we met this issue last time?

[Igor Shishkin (teran)]

Marking invalid according to policy.