Fuel for OpenStack

Creation of neutron ports fails after adding a 9.1 controller to 9.0 cluster

Bug #1628540 reported by Alexander Zatserklyany
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Invalid
High
Nikita Karpin
Fuel for OpenStack 11.0
Mitaka
Invalid
High
Nikita Karpin
Fuel for OpenStack 9.2
Newton
Invalid
High
Sergii Golovatiuk
Fuel for OpenStack 10.1

Bug Description

Steps to reproduce:
-------------------
1. Setup a clear Fuel 9.0
2. Create an local repo on the master node
fuel-mirror create -P ubuntu -G mos
3. Change links for repos mos-updates, mos-security, mos-holdback to
http://<master_ip>:8080/mirrors/mos-repos/ubuntu/9.0
4. Create a cluster with configuration: 3 controllers, 1 compute
5. Deploy the cluster
6. Clone the repo snapshot #315 to the local repo
rsync -az mirror.seed-cz1.fuel-infra.org::mirror/mos-repos/ubuntu/snapshots/9.0-2016-09-23-122322/ /var/www/nailgun/snapshot/
7. Change permission for /var/www/nailgun/snapshot
chown -R root:root /var/www/nailgun/snapshot
chmod -R 755 /var/www/nailgun/snapshot
8. Change URI for repo mos-updates to
deb http://<master_ip>:8080/snapshot mos9.0-proposed main restricted
9. Add a new node with role 'controller'
10. Deploy changes
11. Run OSTF

Expected results:
-----------------
No failed tests

Actual result:
--------------
Launch instance with file injection (failure)

Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0.custom.system_test/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.custom.system_test/fuelweb_test/tests/tests_upgrade/test_add_upgraded_node_to_cluster.py", line 237, in add_updated_controller_to_environment
    self.add_updated_node_to_environment('controller')
  File "/home/jenkins/workspace/9.0.custom.system_test/fuelweb_test/tests/tests_upgrade/test_add_upgraded_node_to_cluster.py", line 192, in add_updated_node_to_environment
    self.fuel_web.run_ostf(cluster_id, test_sets=['smoke', 'sanity'])
  File "/home/jenkins/workspace/9.0.custom.system_test/core/helpers/log_helpers.py", line 32, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.custom.system_test/fuelweb_test/models/fuel_web_client.py", line 1357, in run_ostf
    failed_test_name=failed_test_name, test_sets=test_sets)
  File "/home/jenkins/workspace/9.0.custom.system_test/core/helpers/log_helpers.py", line 32, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0.custom.system_test/fuelweb_test/models/fuel_web_client.py", line 305, in assert_ostf_run
    indent=1)))
AssertionError: Failed 1 OSTF tests; should fail 0 tests. Names of failed tests:
  - Launch instance with file injection (failure) Failed to get to expected status. In error state. Please refer to OpenStack logs for more details.

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

Latest successful test was with snapshot 9.0-2016-09-23-114321

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Revision history for this message
Alexander Kurenyshev (akurenyshev) wrote :

@mos-nova, guys, please take a look on that logs

Changed in fuel:
assignee: nobody → MOS Nova (mos-nova)
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :
Download full text (8.5 KiB)

The issue was reproduced on following environment:
------------------------------------------------
id | name | roles | version
---+---------------------+------------+---------
 3 | slave-05_controller | controller | 9.0
 5 | slave-04_compute | compute | 9.0
 4 | slave-03_controller | controller | 9.1 (snapshot #315)
 1 | slave-02_controller | controller | 9.0
 2 | slave-01_controller | controller | 9.0
 -----------------------------------------------
Test 10 attempt of create-delete server failed
- 6 times on controller node-4 which is 9.1
- 1 time on node-2
- 3 times on node-3
and no fails on node-1

All fails corresponded following records in /var/log/nova/nova-compute.log:
----
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [req-76d93a7e-e469-48a7-9c0e-7efb99a9a291 952683525946431fa2e586a2116c7be9 8e72eafd176a45379104ac98a4fccf69 - - -] [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] Neutron error creating port on network 0fba810c-f21a-4f08-b0ef-f49e4a50b090
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] Traceback (most recent call last):
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 254, in _create_port
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] port = port_client.create_port(port_req_body)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] ret = self.function(instance, *args, **kwargs)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 654, in create_port
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] return self.post(self.ports_path, body=body)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 363, in post
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] headers=headers, params=params)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 298, in do_request
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] self._handle_fault_response(status_code, replybody, resp)
2016-09-28 19:17:12.173 23105 ERROR nova.network.neutronv2.api [instance: 2aa8ba23-6285-4ed4-b25e-f957faab3fa9] File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", li...

Read more...

Changed in fuel:
importance: High → Critical
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

This looks very much like https://bugs.launchpad.net/mos/+bug/1589449 , but it was fixed in 9.0. I wonder if it's a configuration issue and you have DNS extension enabled in Neutron on some controllers, but disabled on others. This has nothing to do with file injection. Also the failure is on the compute side, so it's not about the code on the new controller either (it can be configuration, though)

Please provide your environment for RCA.

Changed in fuel:
status: New → Incomplete
assignee: MOS Nova (mos-nova) → Alexander Zatserklyany (zatserklyany)
Revision history for this message
Dmitry Kalashnik (dkalashnik) wrote :

Env: https://custom-ci.infra.mirantis.net/view/9.0/job/9.0.custom.system_test/1270/console

Also I cant see any significant difference between neutron configs:

[root@nailgun ~]# diff -r -u -I '#.*' node-1 node-5/
diff -r -u -I '#.*' node-1/neutron/neutron.conf node-5/neutron/neutron.conf
--- node-1/neutron/neutron.conf 2016-09-28 17:35:30.000000000 +0000
+++ node-5/neutron/neutron.conf 2016-09-28 17:41:10.000000000 +0000
@@ -9,7 +9,7 @@

 # The host IP to bind to (string value)
 #bind_host = 0.0.0.0
-bind_host = 10.109.26.3
+bind_host = 10.109.26.8

 # The port to bind to (port value)
 # Minimum value: 0
@@ -300,13 +298,6 @@
 #max_allowed_address_pair = 10

 #
-# From neutron.qos
-#
-
-# Drivers list to use to send the update notification (list value)
-#notification_drivers = message_queue
-
-#
 # From oslo.log
 #

@@ -1118,7 +1109,7 @@
 # RabbitMQ HA cluster host:port pairs. (list value)
 # Deprecated group/name - [DEFAULT]/rabbit_hosts
 #rabbit_hosts = $rabbit_host:$rabbit_port
-rabbit_hosts = 10.109.26.3:5673, 10.109.26.4:5673, 10.109.26.8:5673, 10.109.26.2:5673
+rabbit_hosts = 10.109.26.8:5673, 10.109.26.3:5673, 10.109.26.4:5673, 10.109.26.2:5673

 # Connect over SSL for RabbitMQ. (boolean value)
 # Deprecated group/name - [DEFAULT]/rabbit_use_ssl
@@ -1288,6 +1279,16 @@
 #policy_dirs = policy.d

+[qos]
+
+#
+# From neutron.qos
+#
+
+# Drivers list to use to send the update notification (list value)
+#notification_drivers = message_queue
+
+
 [quotas]

 #
diff: node-1/neutron/plugin.ini: No such file or directory
diff: node-5/neutron/plugin.ini: No such file or directory

Changed in fuel:
status: Incomplete → Confirmed
Dmitry Kalashnik (dkalashnik)
Changed in fuel:
assignee: Alexander Zatserklyany (zatserklyany) → Roman Podoliaka (rpodolyaka)
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Please take a look at:

http://paste.openstack.org/show/583430/

"dns-integration" plugin is not enabled on node-3, which confuses nova-compute's: when a neutronclient instance is created the list of supported extensions is cached, thus, nova-compute will send an additional request parameter "dns-name", if "dns-integration" extension is supported.

In your setup haproxy may route it to node-3 which actually does not support "dns-integration" - that causes the errors you see on creation of a new port.

We do not support environments with mixed configuration of extensions in APIs - all neutron-server's must be configured with the very same extensions enabled.

Changed in fuel:
status: Confirmed → Incomplete
assignee: Roman Podoliaka (rpodolyaka) → Alexander Zatserklyany (zatserklyany)
Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

Error in #4
------------------------------------------------
id | name | roles | version
---+---------------------+------------+---------
 3 | slave-05_controller | controller | 9.1 (snapshot #315)
 5 | slave-04_compute | compute | 9.0
 4 | slave-03_controller | controller | 9.0
 1 | slave-02_controller | controller | 9.0
 2 | slave-01_controller | controller | 9.0

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

diff --side-by-side -W 250 --suppress-common-lines node-1.lst node-3.lst
http://paste.openstack.org/show/583434/

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

MOS Neutron, could you please take a look at this one? Looks like configuration in /etc/neutron is exactly the same, the only difference is in the package version. Maybe the list of default API extensions changed?

Changed in fuel:
status: Incomplete → Confirmed
assignee: Alexander Zatserklyany (zatserklyany) → MOS Neutron (mos-neutron)
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

Downgraded to High, retargeted to 9.2 and added release-notes tag. This is not release blocker for 9.1 because the workaround for users who run 9.0 but don't want to update to 9.1 would be to disable updates repository to avoid this scenario. I will follow up with Neutron team to see whether there is better workaround and provide comment for release notes.

Changed in fuel:
importance: Critical → High
milestone: 9.1 → 9.2
tags: added: release-notes
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

According to Sergey Belous it may be caused by the following commit (which we have in 9.1):

rpodolyaka@rpodolyaka-pc:~/src/neutron$ git branch -a --contains e0f46f19f79169eaa63b33745233a4706b870b66
  remotes/gerrit/stable/mitaka
  remotes/mirantis/9.0/mitaka
  remotes/mirantis/stable/mitaka
  remotes/origin/stable/mitaka
rpodolyaka@rpodolyaka-pc:~/src/neutron$ git show e0f46f19f79169eaa63b33745233a4706b870b66
commit e0f46f19f79169eaa63b33745233a4706b870b66
Author: Miguel Lavalle <email address hidden>
Date: Sun May 1 23:51:23 2016 +0000

    Don't load DNS integration in l3_router_plugin

    In this patchset we stop the l3_router_plugin from loading the DNS integration
    extension. The DNS integration extension should only be configured by the
    operator by adding 'dns' to the 'extension_driver' list in ml2_conf.ini

    Change-Id: I1321d9821973fe918005a8c2f3dd751af9affe38
    Closes-Bug: #1574694
    (cherry picked from commit 56962922cc439409bcecd808a1e9bbec2c3756f7)

In other words, this very extension was enabled by chance prior to 9.1. Instead operators are expected to explicitly enable it in ml2_conf.ini. Now that we've fixed this in 9.1, it affects mixed 9.0/9.1 environments.

The workaround is to explicitly enable this extension on all controller nodes.

Revision history for this message
Alexander Zatserklyany (zatserklyany) wrote :

After

root@node-3:~# sed -ir 's/extension_drivers = /extension_drivers = dns,/' /etc/neutron/plugins/ml2/ml2_conf.ini
root@node-3:~# service neutron-server restart

Test 10 attempt of create-delete server passed on all nodes

Roman Podoliaka (rpodolyaka)
summary: - OSTF test 'Launch instance with file injection' failed after add
- controller 9.1 RC2 to cluster 9.0
+ Creation of neutron ports fails after adding a 9.1 controller to 9.0
+ cluster
Olga Gusarenko (ogusarenko)
tags: added: release-notes-done
removed: release-notes
Revision history for this message
ElenaRossokhina (esolomina) wrote :

Similar issue on ci: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.add_updated_node_to_environment/8/testReport/(root)/add_updated_controller_to_environment/
instance creation failed, in nova log:
<179>Oct 13 00:30:20 node-5 nova-compute: 2016-10-13 00:30:20.239 27642 ERROR nova.network.neutronv2.api [req-ffb99d87-a23f-4e9b-b3a4-39ebdfdc2339 d3d8ad7adc3942429a5b99ca339ab561 707afe512caf4e92af926a1eb3ad689f - - -] [instance: 558e15fa-c97e-4877-8a3e-7a47c3fb835b] Neutron error creating port on network 5931c9e3-719e-4974-b70c-f5d80005e942
9.2 snapshot #382
Logs: https://drive.google.com/open?id=0B2ag_Bf-ShtTRzVyLUE2bjRWdTA

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/390847

Changed in fuel:
assignee: MOS Neutron (mos-neutron) → Nikita Karpin (mkarpin)
status: Confirmed → In Progress
Bogdan Dobrelya (bogdando)
Changed in fuel:
milestone: 9.2 → 11.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/390847
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4fb720b01855b00c5f411e928639454adf5851cb
Submitter: Jenkins
Branch: master

commit 4fb720b01855b00c5f411e928639454adf5851cb
Author: Mykyta Karpin <email address hidden>
Date: Wed Oct 26 15:36:45 2016 +0300

    Set dns extension driver

    Dns-integration extension was disabled by default in neutron,
    we must set it explisitly

    Change-Id: I7a9aae67a6f567f9a6049bcdf96195775de8a16b
    Closes-Bug: #1628540

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391456

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/391467

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/391456
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=379b154bd7e99a7dd0932032a75ee8b634841066
Submitter: Jenkins
Branch: stable/newton

commit 379b154bd7e99a7dd0932032a75ee8b634841066
Author: Mykyta Karpin <email address hidden>
Date: Wed Oct 26 15:36:45 2016 +0300

    Set dns extension driver

    Dns-integration extension was disabled by default in neutron,
    we must set it explisitly

    Change-Id: I7a9aae67a6f567f9a6049bcdf96195775de8a16b
    Closes-Bug: #1628540
    (cherry picked from commit 4fb720b01855b00c5f411e928639454adf5851cb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/391467
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6371f228a3a042a4de5739daa1c5be9334821b99
Submitter: Jenkins
Branch: stable/mitaka

commit 6371f228a3a042a4de5739daa1c5be9334821b99
Author: Mykyta Karpin <email address hidden>
Date: Fri Oct 28 16:01:39 2016 +0300

    Set dns extension driver

    Dns-integration extension was disabled by default in neutron,
    we must set it explisitly

    Change-Id: I7a9aae67a6f567f9a6049bcdf96195775de8a16b
    Closes-Bug: #1628540
    (cherry picked from commit 4fb720b01855b00c5f411e928639454adf5851cb)

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :

This bug has been described in 9.1. Added the release-notes tag to describe the bug in 9.2.

tags: added: release-notes
removed: release-notes-done
Revision history for this message
Vitaly Sedelnik (vsedelnik) wrote :

This issue is not actual for 9.2, because it was decided to publish 9.2 to separate repository so 9.0/9.1 clusters won't get 9.2 packages. No need to verify it in 9.2 cycle.

TatyanaGladysheva (tgladysheva)
tags: removed: on-verification
Revision history for this message
Maria Zlatkova (mzlatkova) wrote :

Not to be described in RN 9.2.

tags: added: release-notes-done
removed: release-notes
Revision history for this message
Sergey Novikov (snovikov) wrote :

According to comment #22 I've moved the bug to "Invalid" state for 9.2 and 10.0

Changed in fuel:
status: Fix Committed → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.