Network verification failed after successfull deployment with an error few nodes failed to connect to some repositories

[Bogdan Dobrelya]

Deployment with 3 controllers, NeutronVLAN before Backup/Restore

Scenario:
1. Setup master node
2. Create new environment
3. Choose Neutron, VLAN
4. Add 3 controller
5. Add 2 compute
6. Verify networks (I skipped this step)
7. Deploy the environment
8. Verify networks (failure was here)

Expected:
Passed verification at step 8

Actual:
Verification failed.
These nodes: "4", "3" failed to connect to some of these repositories: "http://archive.ubuntu.com/ubuntu/", "http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/"

api: '1.0'
astute_sha: 53c86cba593ddbac776ce5a3360240274c20738c
auth_required: true
build_id: 2015-08-26_02-59-52
build_number: '234'
feature_groups:
- mirantis
fuel-agent_sha: 4c2ab9d6c623d345086c6e2874d1df81fd96a942
fuel-library_sha: a0e2b70120d465b0f8816d56e7b0833fd24d3f0c
fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
fuel-ostf_sha: dec66ae87b3e5f8b84670c6b4dbd184b0a3c7770
fuelmain_sha: da38cc137df3b59015f408f8a401fdb6b428f3c0
nailgun_sha: fceb77659d76766a05548696f1e9bf4ad25d39ba
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: 9643fa07f1290071511066804f962f62fe27b512
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 53c86cba593ddbac776ce5a3360240274c20738c
      build_id: 2015-08-26_02-59-52
      build_number: '234'
      feature_groups:
      - mirantis
      fuel-agent_sha: 4c2ab9d6c623d345086c6e2874d1df81fd96a942
      fuel-library_sha: a0e2b70120d465b0f8816d56e7b0833fd24d3f0c
      fuel-nailgun-agent_sha: e01693992d7a0304d926b922b43f3b747c35964c
      fuel-ostf_sha: dec66ae87b3e5f8b84670c6b4dbd184b0a3c7770
      fuelmain_sha: da38cc137df3b59015f408f8a401fdb6b428f3c0
      nailgun_sha: fceb77659d76766a05548696f1e9bf4ad25d39ba
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: 9643fa07f1290071511066804f962f62fe27b512
      release: '7.0'

[Alexander Chudnovets (achudnovets)]

Can't reproduce. Tried on builds 224 and 234

[Alexander Kurenyshev (akurenyshev)]

Another reproduce on the CI job: https://product-ci.infra.mirantis.net/job/7.0.system_test.ubuntu.thread_non_func_1/68/console

Steps are used in test:
1. Create cluster
2 Configure external NTP,DNS settings
3. Add 3 nodes with controller roles
4. Add 2 nodes with compute roles
5. Deploy the cluster
6. Run Net Check

Expected result:
Net check passes

Actual result:
net check fails with an error:
Task 'verify_networks' has incorrect status. error != ready, 'These nodes: "5", "3", "1", "4", "2" failed to connect to some of these repositories: "http://osci-mirror-msk.msk.mirantis.net/pkgs/ubuntu-2015-09-01-170158/"'

Full trace:
======================================================================
FAIL: Use external ntp and dns in ha mode
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/helpers/decorators.py", line 80, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/tests/tests_strength/test_master_node_failover.py", line 189, in deploy_ha_dns_ntp
    self.fuel_web.verify_network(cluster_id)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/helpers/decorators.py", line 362, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/models/fuel_web_client.py", line 1407, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/7.0.system_test.ubuntu.thread_non_func_1/fuelweb_test/models/fuel_web_client.py", line 306, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'verify_networks' has incorrect status. error != ready, 'These nodes: "5", "3", "1", "4", "2" failed to connect to some of these repositories: "http://osci-mirror-msk.msk.mirantis.net/pkgs/ubuntu-2015-09-01-170158/"'

Fuel used:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "262"
  build_id: "262"
  nailgun_sha: "b564ae20116297750bf6402b3a017e219bf4b468"
  python-fuelclient_sha: "9643fa07f1290071511066804f962f62fe27b512"
  fuel-agent_sha: "082a47bf...

[Dmitry Pyzhov (dpyzhov)]

Looks like issue with access to external networks form nodes. What happens when you rerun network checks?

[Dmitry Pyzhov (dpyzhov)]

Igor, do we have retries in this check? Is it clear in logs that we tried to connect several times and failed?

[Ihor Kalnytskyi (ikalnytskyi)]

Dmitry,

Unfortunately, the patch was merged only in master, and we have decided to go without backport.

[Ihor Kalnytskyi (ikalnytskyi)]

Move it to Incomplete. Please try to reproduce it on master iso, and provide logs. It should help us to detect who's responsible for failures.

[Tatyanka (tatyana-leontovich)]

Not reproduced for the long time, so move to invalid