Task[connectivity_tests/] failed due to mirror connection issue

[Anton Chevychalov]

Failed BVT:
===========
https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/741/

Failed task:
============
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[connectivity_tests/3], Task[connectivity_tests/5], Task[connectivity_tests/6] Stopping the deployment process!'

Astuted.log
===========
2017-01-09 07:28:45 INFO [29312] Casting message to Nailgun:
{"method"=>"deploy_resp",
 "args"=>
  {"task_uuid"=>"5c75c8c9-c6ad-4520-b01f-559eed992feb",
   "status"=>"error",
   "progress"=>100,
   "error"=>
    "All nodes are finished. Failed tasks: Task[connectivity_tests/3], Task[connectivity_tests/5], Task[connectivity_tests/6] Stopping the deployment process!"}}

Puppet.log on node-6:
=====================

2017-01-09 07:24:43 +0000 Puppet (err): ERROR: Unable to fetch url 'http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/ubuntu-2017-01-09-030059/', error 'execution expired'. Please verify node connectivity to t
his URL, or remove it from the settings page if it is invalid. on node node-6.test.domain.local
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:95:in `rescue in fetch'
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:83:in `fetch'
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:106:in `block (4 levels) in <top (required)>'

Possible reason:
================
Mirror was not in reachable state.

[Nastya Urlapova (aurlapova)]

Compute nodes do not have connectivity to repos.

[Nastya Urlapova (aurlapova)]

Repos are available on master and controller nodes.

[Georgy Kibardin (gkibardin)]

The reason is that on nodes where there is no br-ex interface default route is setup via br-mgmt for which there is no NAT to the external network.

[Vladimir Kuklin (vkuklin)]

This is an expected configuration by default since 6.x releases. The default route for compute nodes is configured through vrouter that is a network namespace called vrouter which is started on one of the controller nodes. NAT is configured within this namespace, so I would look into forwarding or networking config on the host node for tests.

[Vladimir Kuklin (vkuklin)]

Moreover, the nodes that have connectivity check failing are controller nodes which actually have br-ex interface

[Georgy Kibardin (gkibardin)]

All failing nodes are computes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/417952

[Vladimir Kuklin (vkuklin)]

Okay, I found the issue - it was the regression as we flush default route that is set by vip__vrouter_pub resource

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/417968

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417969

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417968
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9923c0970f64a7299a5697a9a850e48b4fac8f91
Submitter: Jenkins
Branch: stable/mitaka

commit 9923c0970f64a7299a5697a9a850e48b4fac8f91
Author: Vladimir Kuklin <email address hidden>
Date: Mon Jan 9 19:22:13 2017 +0300

    Fix vrouter route flush

    1. Do not set routing on status command - this is useless and
    destructive

    2. Save default routes into a separate file and restore them
    after flush

    Change-Id: Ia128979920e054343b2ac05e437683772c81731a
    Closes-bug: #1654967

[Ekaterina Shutova (eshutova)]

Verified on 9.2 snapshot #742.
Latest runs with fix are passed: https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/745/testReport/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/418934

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/418935

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/418937

[Nastya Urlapova (aurlapova)]

New failures on 9.2 snapshot #748:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.env_customizations_check/104/testReport/(root)/customizations_check_env/customizations_check_env/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bvt_ubuntu_bootstrap/172/testReport/(root)/reset_on_ready_ubuntu_bootstrap/reset_on_ready_ubuntu_bootstrap/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bvt_ubuntu_bootstrap/172/testReport/(root)/deploy_with_two_ubuntu_bootstraps/deploy_with_two_ubuntu_bootstraps/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.services_reconfiguration_thread_2/169/testReport/(root)/two_clusters_reconfiguration/two_clusters_reconfiguration/
Dima Ukov found error in logs:
>>>
2017-01-12 01:44:30 +0000 Scope(Class[Osnailyfacter::Netconfig::Connectivity_tests]) (notice): MODULAR: netconfig/connectivity_tests.pp
2017-01-12 01:45:30 +0000 Puppet (err): ERROR: Unable to fetch url 'http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/ubuntu-2017-01-11-030059/', error 'execution expired'. Please verify node connectivity to this URL, or remove it from the settings page if it is invalid. on node node-2.test.domain.local
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:95:in `rescue in fetch'
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:83:in `fetch'
/etc/puppet/modules/osnailyfacter/lib/puppet/parser/functions/url_available.rb:106:in `block (4 levels) in <top (required)>'
2017-01-12 01:45:30 +0000 Puppet (err): ERROR: Unable to fetch url 'http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/ubuntu-2017-01-11-030059/', error 'execution expired'. Please verify node connectivity to this URL, or remove it from the settings page if it is invalid.

[Alexander Kurenyshev (akurenyshev)]

And again https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.full_cluster_reinstallation/169/console
Test Verify full cluster reinstallation.
Error:AssertionError: Task 'deploy' has incorrect status. error != ready, 'All nodes are finished. Failed tasks: Task[connectivity_tests/1] Stopping the deployment process!'
Verification failed.
Repo availability verification failed on following nodes slave-04_compute_cinder.
Following repos are not available - http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/ubuntu-2017-01-11-030059/, http://mirror.fuel-infra.org/mos-repos/ubuntu/9.2, http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2017-01-11-172422.

[Alexander Kurenyshev (akurenyshev)]

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.unlock_settings_tab_states/169/console
Check settings and network tabs is unlocked for a positively
deployed and redeployed cluster
Error: AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[top-role-compute/3], Task[top-role-compute/5] Stopping the deployment process!'
Because in puppet:
Err http://mirror.seed-cz1.fuel-infra.org/pkgs/snapshots/ubuntu-2017-01-11-030059/ trusty/main libasound2-data all 1.0.27.2-3ubuntu7
  Could not connect to mirror.seed-cz1.fuel-infra.org:80 (5.43.231.83), connection timed out

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417969
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=59909bda6c65f0f810fd3f8b76a8e19f1e2d3934
Submitter: Jenkins
Branch: stable/newton

commit 59909bda6c65f0f810fd3f8b76a8e19f1e2d3934
Author: Vladimir Kuklin <email address hidden>
Date: Mon Jan 9 19:22:13 2017 +0300

    Fix vrouter route flush

    1. Do not set routing on status command - this is useless and
    destructive

    2. Save default routes into a separate file and restore them
    after flush

    Change-Id: Ia128979920e054343b2ac05e437683772c81731a
    Closes-bug: #1654967

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/417952
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=9f3edd83bb8686c395e4437351c110e2e1e3f511
Submitter: Jenkins
Branch: master

commit 9f3edd83bb8686c395e4437351c110e2e1e3f511
Author: Vladimir Kuklin <email address hidden>
Date: Mon Jan 9 19:22:13 2017 +0300

    Fix vrouter route flush

    1. Do not set routing on status command - this is useless and
    destructive

    2. Save default routes into a separate file and restore them
    after flush

    Change-Id: Ia128979920e054343b2ac05e437683772c81731a
    Closes-bug: #1654967

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418934
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=066c7e5de99bce37db6b532ca4325e202e544be2
Submitter: Jenkins
Branch: master

commit 066c7e5de99bce37db6b532ca4325e202e544be2
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 11 16:39:16 2017 +0300

    Set ip_forward in vrouter namespace to 1 each monitor

    For some reason we can have ip_forward reset sometimes
    in the vrouter namespace, which leads to connectivity tests
    issues. In order to fix this we move ip_forward setting
    to get_ns() function to monitor and set it each monitor operation

    Change-Id: I2d10d465259adc1f30161fee488feef4179b5c70
    Related-bug: #1654967

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418935
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=aea46cf7ba5b99824cc858d30b0b26d3dfbb5ec6
Submitter: Jenkins
Branch: stable/mitaka

commit aea46cf7ba5b99824cc858d30b0b26d3dfbb5ec6
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 11 16:39:16 2017 +0300

    Set ip_forward in vrouter namespace to 1 each monitor

    For some reason we can have ip_forward reset sometimes
    in the vrouter namespace, which leads to connectivity tests
    issues. In order to fix this we move ip_forward setting
    to get_ns() function to monitor and set it each monitor operation

    Change-Id: I2d10d465259adc1f30161fee488feef4179b5c70
    Related-bug: #1654967

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/418937
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ccd31901ad22ffd52aaaa15b92e678e933ee91e5
Submitter: Jenkins
Branch: stable/newton

commit ccd31901ad22ffd52aaaa15b92e678e933ee91e5
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 11 16:39:16 2017 +0300

    Set ip_forward in vrouter namespace to 1 each monitor

    For some reason we can have ip_forward reset sometimes
    in the vrouter namespace, which leads to connectivity tests
    issues. In order to fix this we move ip_forward setting
    to get_ns() function to monitor and set it each monitor operation

    Change-Id: I2d10d465259adc1f30161fee488feef4179b5c70
    Related-bug: #1654967

[Dmitry Belyaninov (dbelyaninov)]

Verified on 9.2 snapshot #771.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[Ilya Bumarskov (ibumarskov)]

Verified on Fuel 10 build# 1483:
https://product-ci.infra.mirantis.net/job/10.0.main.ubuntu.bvt_2/1397