Fuel for OpenStack

Network configuration changed during the deployment

Bug #1543535 reported by Timur Nurlygayanov on 2016-02-09

This bug report is a duplicate of: Bug #1532823: Network verification has failed after successful HA deployment. Second controller has changed network configuration after deploy (not idempotent). Edit Remove

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	Confirmed	Undecided	Timur Nurlygayanov	Fuel for OpenStack 9.0
	8.0.x	Confirmed	Medium	Fuel Python (Deprecated)	Fuel for OpenStack 8.0

Bug Description

This issue reproduced not is 100% of cases, and it is probably regression issue.
It is Critical because sometimes fuel can randomly change the mapping of ETH interfaces with logical networks and this will lead to fail of deployments / networks verification after the deployment.

This issue was found by MOS QA automated test suite (Neutron destructive suite)
Template which was used for deployment: https://github.com/Mirantis/mos-ci-deployment-scripts/blob/master/3_controllers_2compute_neutron_env_template.yaml
and https://github.com/Mirantis/mos-ci-deployment-scripts/blob/master/mos_tests.yaml

We used fuel-qa code from commit 87df82b0a7d204d2b43025f50ad232997bf0a35b (master branch)

Steps To Reproduce:
1. Run fuel master node and 3 slave nodes
2. Configure environment with 3 controllers, 3 computes, Neutron VxVLAN, L2pop and L3 HA features enabled
3. Check networks configuration and make sure that everything works fine
4. Deploy the environment
5. Start networks check one more time after the deployment

Expected Result:
Deployment will pass, network connectivity checks will pass as well

Observed Result:
Deployment passed (in my case, it depends on the interfaces which will be affected be the issue during the deployment, in fact, it can fail the deployment)
Network verification fails with the error:
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

Diagnostic snapshot and screenshots are attached.

See original description

Tags:

Timur Nurlygayanov (tnurlygayanov) on 2016-02-09

description:

updated

Timur Nurlygayanov (tnurlygayanov) on 2016-02-09

Changed in fuel:
milestone:	none → 9.0
importance:	Undecided → Critical

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-09:

Download full text (17.9 KiB)

Logs from jenkins:

<<< --------------------------------------[ FINISH Step 004. Create Fuel Environment STEP TOOK 0 min 7 sec ]-------------------------------------- >>>

ok
Add nodes to environment ... 2016-02-09 01:06:29,482 - INFO decorators.py:53 --
<<< -------------------------------------------------[ START Step 005. Add nodes to environment ]------------------------------------------------- >>>

2016-02-09 01:06:29,483 - INFO actions_base.py:186 -- Add nodes to env 1
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-01
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-02
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-03
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['compute'] to node slave-04
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['compute'] to node slave-05
2016-02-09 01:06:31,023 - INFO fuel_web_client.py:1391 -- Assigned networks are: {'enp0s6': ['private'], 'enp0s7': ['storage'], 'enp0s4': ['public'], 'enp0s5': ['management'], 'enp0s3': ['fuelweb_admin']}
2016-02-09 01:06:31,600 - INFO decorators.py:61 --
<<< -------------------------------------[ FINISH Step 005. Add nodes to environment STEP TOOK 0 min 2 sec ]-------------------------------------- >>>

ok
Run network checker ... 2016-02-09 01:06:31,672 - INFO decorators.py:53 --
<<< ---------------------------------------------------[ START Step 006. Run network checker ]---------------------------------------------------- >>>

2016-02-09 01:06:31,673 - INFO fuel_web_client.py:1044 -- Run network verification on the cluster 1
2016-02-09 01:07:32,483 - INFO fuel_web_client.py:1358 -- Network verification of cluster 1 finished
2016-02-09 01:07:32,483 - INFO decorators.py:61 --
<<< ----------------------------------------[ FINISH Step 006. Run network checker STEP TOOK 1 min 1 sec ]---------------------------------------- >>>

ok
Deploy environment ... 2016-02-09 01:07:32,495 - INFO decorators.py:53 --
<<< ----------------------------------------------------[ START Step 007. Deploy environment ]---------------------------------------------------- >>>

Logs from jenkins:

<<< --------------------------------------[ FINISH Step 004. Create Fuel Environment STEP TOOK 0 min 7 sec ]-------------------------------------- >>>

ok
Add nodes to environment ... 2016-02-09 01:06:29,482 - INFO decorators.py:53 -- 
<<< -------------------------------------------------[ START Step 005. Add nodes to environment ]------------------------------------------------- >>>

2016-02-09 01:06:29,483 - INFO actions_base.py:186 -- Add nodes to env 1
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-01
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-02
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['controller', 'cinder'] to node slave-03
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['compute'] to node slave-04
2016-02-09 01:06:29,483 - INFO actions_base.py:200 -- Set roles ['compute'] to node slave-05
2016-02-09 01:06:31,023 - INFO fuel_web_client.py:1391 -- Assigned networks are: {'enp0s6': ['private'], 'enp0s7': ['storage'], 'enp0s4': ['public'], 'enp0s5': ['management'], 'enp0s3': ['fuelweb_admin']}
2016-02-09 01:06:31,600 - INFO decorators.py:61 -- 
<<< -------------------------------------[ FINISH Step 005. Add nodes to environment STEP TOOK 0 min 2 sec ]-------------------------------------- >>>

ok
Run network checker ... 2016-02-09 01:06:31,672 - INFO decorators.py:53 -- 
<<< ---------------------------------------------------[ START Step 006. Run network checker ]---------------------------------------------------- >>>

2016-02-09 01:06:31,673 - INFO fuel_web_client.py:1044 -- Run network verification on the cluster 1
2016-02-09 01:07:32,483 - INFO fuel_web_client.py:1358 -- Network verification of cluster 1 finished
2016-02-09 01:07:32,483 - INFO decorators.py:61 -- 
<<< ----------------------------------------[ FINISH Step 006. Run network checker STEP TOOK 1 min 1 sec ]---------------------------------------- >>>

ok
Deploy environment ... 2016-02-09 01:07:32,495 - INFO decorators.py:53 -- 
<<< ----------------------------------------------------[ START Step 007. Deploy environment ]---------------------------------------------------- >>>

2016-02-09 01:07:32,495 - INFO fuel_web_client.py:741 -- Deploy cluster 1
2016-02-09 01:07:32,495 - INFO fuel_web_client.py:793 -- Launch deployment of a cluster #1
2016-02-09 01:07:36,204 - INFO fuel_web_client.py:315 -- Assert task {u'status': u'pending', u'name': u'deploy', u'cluster': 1, u'result': {}, u'progress': 0, u'message': None, u'id': 5, u'uuid': u'cd2c26f2-dd11-43db-b4b2-2ff569ea8d42'} is success
2016-02-09 01:07:36,204 - INFO fuel_web_client.py:1104 -- Wait for task 10000 seconds: 
 status       pending
 name         deploy
 cluster      1
 result       {}
 progress     0
 message      None
 id           5
 uuid         cd2c26f2-dd11-43db-b4b2-2ff569ea8d42
2016-02-09 02:46:44,996 - INFO fuel_web_client.py:1121 -- Task finished. Took 5948.77952504 seconds. 
 status       ready
 name         deploy
 cluster      1
 result       {}
 progress     100
 message      Provision of environment 'mos-tests' is done. 
Deployment of environment 'mos-tests' is done. 
 id           5
 uuid         cd2c26f2-dd11-43db-b4b2-2ff569ea8d42
2016-02-09 02:46:45,008 - INFO fuel_web_client.py:161 -- Waiting 1200 sec. for passed OSTF HA tests.
2016-02-09 02:48:32,045 - INFO fuel_web_client.py:167 -- OSTF HA tests passed successfully.
2016-02-09 02:48:32,046 - INFO fuel_web_client.py:178 -- Waiting 300 sec. for passed OSTF Sanity checks.
2016-02-09 02:49:28,557 - INFO fuel_web_client.py:184 -- OSTF Sanity checks passed successfully.
2016-02-09 02:49:29,958 - INFO fuel_web_client.py:784 -- Node status: 
 slave-05:
    Host       node-1
    Roles:
       Hiera:
           - compute
       Nailgun:
           - compute
    Memory:
       RAM:
          used        1718
          cached      1161
          free        2234
          shared      0
          total       3953
          buffers     128
       SWAP:
          total     4095
          free      4095
          used      0
2016-02-09 02:49:31,420 - INFO fuel_web_client.py:784 -- Node status: 
 slave-03:
    Host       node-4
    Roles:
       Hiera:
           - cinder
           - controller
       Nailgun:
           - controller
           - cinder
    Memory:
       RAM:
          used        3728
          cached      293
          free        224
          shared      44
          total       3953
          buffers     14
       SWAP:
          total     4095
          free      4094
          used      1
2016-02-09 02:49:33,277 - INFO fuel_web_client.py:784 -- Node status: 
 slave-02:
    Host       node-5
    Roles:
       Hiera:
           - cinder
           - controller
       Nailgun:
           - controller
           - cinder
    Memory:
       RAM:
          used        3825
          cached      358
          free        128
          shared      44
          total       3953
          buffers     23
       SWAP:
          total     4095
          free      4094
          used      1
2016-02-09 02:49:34,574 - INFO fuel_web_client.py:784 -- Node status: 
 slave-04:
    Host       node-2
    Roles:
       Hiera:
           - compute
       Nailgun:
           - compute
    Memory:
       RAM:
          used        1718
          cached      1162
          free        2234
          shared      0
          total       3953
          buffers     128
       SWAP:
          total     4095
          free      4095
          used      0
2016-02-09 02:49:36,034 - INFO fuel_web_client.py:784 -- Node status: 
 slave-01:
    Host       node-3
    Roles:
       Hiera:
           - cinder
           - primary-controller
       Nailgun:
           - controller
           - cinder
    Memory:
       RAM:
          used        3767
          cached      192
          free        186
          shared      60
          total       3953
          buffers     6
       SWAP:
          total     4095
          free      4094
          used      1
2016-02-09 02:49:36,035 - INFO fuel_web_client.py:856 -- Get ID of a last created cluster
2016-02-09 02:50:26,162 - INFO fuel_web_client.py:856 -- Get ID of a last created cluster
2016-02-09 02:50:31,402 - INFO decorators.py:61 -- 
<<< ---------------------------------------[ FINISH Step 007. Deploy environment STEP TOOK 102 min 59 sec ]--------------------------------------- >>>

ok
Run network checker ... 2016-02-09 02:50:32,872 - INFO decorators.py:53 -- 
<<< ---------------------------------------------------[ START Step 008. Run network checker ]---------------------------------------------------- >>>

2016-02-09 02:50:32,873 - INFO fuel_web_client.py:1044 -- Run network verification on the cluster 1
2016-02-09 02:50:33,369 - ERROR __init__.py:67 -- assert_task_success raised: AssertionError('Task \'check_networks\' has incorrect status. error != ready, \'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:\n"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"\'',)
Traceback: Traceback (most recent call last):
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

2016-02-09 02:50:33,381 - ERROR fuel_web_client.py:1342 -- Network verification failed:
Node Name                      | Node MAC address     | Node Interface  | Expected VLAN (not received)
-                              | -                    | -               | []
Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"
2016-02-09 02:50:33,382 - ERROR __init__.py:67 -- verify_network raised: AssertionError('Task \'check_networks\' has incorrect status. error != ready, \'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:\n"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"\'',)
Traceback: Traceback (most recent call last):
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 1354, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

2016-02-09 02:50:53,399 - INFO fuel_web_client.py:1044 -- Run network verification on the cluster 1
2016-02-09 02:50:53,763 - ERROR __init__.py:67 -- assert_task_success raised: AssertionError('Task \'check_networks\' has incorrect status. error != ready, \'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:\n"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"\'',)
Traceback: Traceback (most recent call last):
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

2016-02-09 02:50:53,777 - ERROR fuel_web_client.py:1342 -- Network verification failed:
Node Name                      | Node MAC address     | Node Interface  | Expected VLAN (not received)
-                              | -                    | -               | []
Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"
2016-02-09 02:50:53,777 - ERROR __init__.py:67 -- verify_network raised: AssertionError('Task \'check_networks\' has incorrect status. error != ready, \'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:\n"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"\'',)
Traceback: Traceback (most recent call last):
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 1354, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal
    raise ASSERTION_ERROR(message)
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

2016-02-09 02:50:53,851 - INFO fuel_web_client.py:1104 -- Wait for task 600 seconds: 
 status       running
 name         dump
 cluster      None
 result       {}
 progress     0
 message      None
 id           13
 uuid         d9fa8b8e-2f0b-4c63-ad86-c8ffd1685269
2016-02-09 02:53:49,567 - INFO fuel_web_client.py:1121 -- Task finished. Took 175.704446077 seconds. 
 status       ready
 name         dump
 cluster      None
 result       {}
 progress     100
 message      /api/dump/fuel-snapshot-2016-02-08_23-50-54.tar.xz
 id           13
 uuid         d9fa8b8e-2f0b-4c63-ad86-c8ffd1685269
2016-02-09 02:53:49,577 - INFO decorators.py:49 -- Saving logs to "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/logs/fail_error_Case_CreateDeployOstf__Config_3_controllers_2compute_neutron_env.Step008_network_check-fuel-snapshot-2016-02-08_23-50-54.tar.xz" file
2016-02-09 02:55:19,258 - INFO decorators.py:299 -- <<<<<****************************************************************************************************>>>>>
2016-02-09 02:55:19,258 - INFO decorators.py:300 -- Failed in method 'Case_CreateDeployOstf__Config_3_controllers_2compute_neutron_env.Step008_network_check'. Make snapshot: CreateDeployOstf_ff1c06b0_Step008
2016-02-09 02:55:19,259 - INFO decorators.py:310 -- You could revert and ssh to master node: [source /root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/bin/activate; dos.py revert-resume MOS_CI_fuel-8.0-541-2016-02-08_13-55-16.isoL3_HA_2flow CreateDeployOstf_ff1c06b0_Step008 && ssh root@10.109.8.2]
2016-02-09 02:55:19,259 - INFO decorators.py:312 -- <<<<<****************************************************************************************************>>>>>
FAIL
Run health checker ... SKIP: Failure in <function Case_CreateDeployOstf__Config_3_controllers_2compute_neutron_env.Step008_network_check at 0x7f17cc8bf1b8>
Finish test case ... 2016-02-09 02:55:20,390 - INFO actions_base.py:90 -- 
<<< =========================================[ FINISH Case deploy Environment CASE TOOK 141 min 26 sec ]========================================== >>>

======================================================================
FAIL: Run network checker
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-devops-venv/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/system_test/helpers/decorators.py", line 54, in wrapper
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/system_test/helpers/decorators.py", line 79, in wrapper
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/system_test/tests/actions_base.py", line 310, in network_check
    self.fuel_web.verify_network(self.cluster_id)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/helpers/decorators.py", line 335, in wrapper
    return func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 1354, in verify_network
    self.assert_task_success(task, timeout, interval=10)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/root/.jenkins/jobs/create_param_environment/workspace@2/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'check_networks' has incorrect status. error != ready, 'Some untagged networks are assigned to the same physical interface. You should assign them to different physical interfaces. Affected:
"admin (PXE)", "storage" networks at node "slave-03_controller_cinder"'

----------------------------------------------------------------------
Ran 12 tests in 8486.113s

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-09:

fuel-snapshot-2016-02-09_11-36-23.tar.xz Edit (102.5 MiB, application/octet-stream)

Diagnostic snapshot:

Changed in fuel:
assignee:	nobody → Dennis Dmitriev (ddmitriev)

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-09:

Screenshot from 2016-02-09 14:24:17.png Edit (139.3 KiB, image/png)

Screenshot of network interfaces for controller-3

Other controllers configured correctly.

description:	updated
tags:	added: area-library area-qa system-tests

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-09:

QA framework team is investigating the issue, it can be the issue in fuel-qa code but it is also can be the regression issue in fuel library / nailgun / astute.

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2016-02-09:

Timur your snapshot is missing docker-logs, so could you please attach it

Changed in fuel:
status:	New → Incomplete
importance:	Critical → Undecided
assignee:	Dennis Dmitriev (ddmitriev) → Timur Nurlygayanov (tnurlygayanov)

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2016-02-09:

Looks like it is the product bug.

There is the data that was put to the Nailgun for node slave-03 configuration (node id=4): http://paste.openstack.org/show/486390/

There is the result after network verification: http://paste.openstack.org/show/486397/

Somehow, the network 'fuelweb_admin' was moved from the interface 'enp0s3' to the interface 'enp0s7'.

Thanks to @Tatyanka for the bug that was already filed for this issue: https://bugs.launchpad.net/fuel/+bug/1532823 and was fixed for ISO #506.

But here we have a reproduce on the ISO #541 :
[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "541"
  build_id: "541"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "87dfb6bc25d4650264f09c338ed77c21a3d6fe87"

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-09:

fuel-snapshot-2016-02-09_11-36-23.tar.xz Edit (102.5 MiB, application/octet-stream)

Here are logs from docker-logs folder ^^

Denis checked the logs and found that fuel-qa code sent correct information about the configuration of networks interfaces.
This is an issue with Nailgun / Astute / fuel library.

Status changed to Confirmed, priority changed to Critical because if we will randomly change the network interfaces mapping we will fail deployments. This issue reproduced on virtual lab with 6 VMs, it can be easily reproduced on large deployments (many nodes - many chances to randomly change an interface on one of them).

The possible bad workaround is to redeploy all nodes which will have issues, but we can guarantee that users will not spend several days trying to deploy environments then.

Revision history for this message

Dennis Dmitriev (ddmitriev) wrote on 2016-02-09:

Please do not mark this bug as a duplicate of #1532823.

#1532823 is already fixed, and here we have a regression that should be fixed separatelly.

tags:

added: area-python
removed: area-library area-qa system-tests

Revision history for this message

Tatyanka (tatyana-leontovich) wrote on 2016-02-09:

Actually you are wrong, it is not regression, it is mean that previous issue is partly fixed - so move to duplicate

Timur Nurlygayanov (tnurlygayanov) on 2016-02-09

Changed in fuel:
status:	Incomplete → Confirmed

Revision history for this message

Timur Nurlygayanov (tnurlygayanov) wrote on 2016-02-11:

#10

docker-logs.tar.gz Edit (31.2 MiB, application/x-tar)

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1532823 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.