Neutron services are not restarted after config changes in common config sections

[Ilya Bumarskov]

Fuel 9.1 (snapshot-id#138) with DVS plugin 3.0.0

Detailed bug description:
OSTF failed on step "Create instance".

Steps to reproduce:
   - Create env with vCenter and DVS plugin
   - Add following nodes:
         * Controller
         * Compute + Cinder
         * Cinder-vmware
         * Compute-vmware
   - Configure VMware vCenter Settings. Add 2 vSphere clusters and configure Nova Compute instances on controller and compute-vmware.
   - Deploy env
   - Add two nodes with controller role.
   - Deploy changes
   - Shutdown primary controller
   - Run OSTF

Observed behavior:
Instance in nova az can't be created.
Nova-compute logs:

nova.compute.manager [req-a6012749-c2bd-4caa-b7e5-dda2dcb40d34 629f4e545c0e499c8450c890439e6e1f 58c4468a004e44888bf6723925513638 - - -] Instance failed network setup after 1 attempt(s) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager Traceback (most recent call last): 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager bind_host_id=bind_host_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 668, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self._delete_ports(neutron, instance, created_port_ids) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager self.force_reraise() 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 657, in allocate_for_instance 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager security_group_ids, available_macs, dhcp_opts) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 259, in _create_port 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager raise exception.PortBindingFailed(port_id=port_id) 2016-08-16 13:06:05.155 18826 ERROR nova.compute.manager PortBindingFailed: Binding failed for port bce50f48-71f5-4044-ad1a-89f2301aeb16,

Neutron-server logs:

neutron.plugins.ml2.managers [req-0442810d-f7a2-4260-9183-0a7788cee79d 39c82b217fc5421ca668297e97b0aed8 d5f913b567b84343bcd98abe18ea3603 - - -] Failed to bind port bce50f48-71f5-4044-ad1a-89f2301aeb16 on host node-1.test.domain.local for vnic_type normal using segments [{'segmentation_id': 1000L, 'physical_network': u'physnet2', 'id': u'5bbb93c3-bc9e-4094-ac1c-114ea342046f', 'network_type': u'vlan'}]

neutron agent-list: http://paste.openstack.org/show/558392/

[Oleksiy Molchanov (omolchanov)]

Please provide diagnostic snapshot, marking as Incomplete.

[Vyacheslav Tabolin (slavchick)]

root@node-1:~# cat /var/log/neutron/neutron-openvswitch-agent.log | grep 'Logging enabled!'
2016-08-16 10:02:28.833 24060 INFO neutron.common.config [-] Logging enabled!

It means that neutron-openvswitch-agent never restarted. It can't reach message queue because agent takes conf params at start only.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/414930

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/420047

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/420054

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414930
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b4f804ea4cb96c3dd8c3d7c48d77464b87213bba
Submitter: Jenkins
Branch: master

commit b4f804ea4cb96c3dd8c3d7c48d77464b87213bba
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420054
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0ecf4ef3282f4785c39c6d6f64dd88484264c89c
Submitter: Jenkins
Branch: stable/mitaka

commit 0ecf4ef3282f4785c39c6d6f64dd88484264c89c
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    (cherry-picked from 5912ee7ea5715f17f2188c3eb9c814867c9af571)
    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/420047
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=73599f5f44f6849c0d47639124a091824a21a543
Submitter: Jenkins
Branch: stable/newton

commit 73599f5f44f6849c0d47639124a091824a21a543
Author: Igor Gajsin <email address hidden>
Date: Mon Dec 26 13:32:38 2016 +0300

    Restart neutron services on neutron config change

    There are many neutron tasks, e.g.: common-config and network-plugins-l2.
    Normally the common-config configures neutron.conf and then plugns-l2 starts
    the ovs-agent.

    But there is the wrong behaviour when new controllers are added to an
    environment:
      1. configure and deploy an env.
      2. common-config configures neutron.conf
      3. plugins-l2 starts ovs-agent.
      4. add new controllers to the env and redeploy it.
      5. common-config configures neutron.conf.
      6. plugins-l2 knows nothing about changes in the config.

    So ovs-agent isn't restarted and works with old configuration data.

    This commit makes neutron tasks check md5 hash of neutron.conf and
    restart them when config is changed.

    (cherry-picked from 5912ee7ea5715f17f2188c3eb9c814867c9af571)
    Change-Id: Ia26f4a5105407ce322a5185455093d69a938edfc
    Closes-bug: #1613724

[Ilya Bumarskov (ibumarskov)]

I can't verify it until developers fix the https://bugs.launchpad.net/fuel/+bug/1658952

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/425055

[Igor Gajsin (igajsin)]

there is another bug that reveals in the same deployment scenario: https://bugs.launchpad.net/fuel/+bug/1658952

When it will be fixed the current bug could be tested for moving to the 'fix-released' state.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/425648

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/425649

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425055
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=418b09fc1f9b43f78ff7b40ae31e70784a656524
Submitter: Jenkins
Branch: master

commit 418b09fc1f9b43f78ff7b40ae31e70784a656524
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425649
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=15be3f5f130e81b31b6132ab2835ce7d4906cc18
Submitter: Jenkins
Branch: stable/newton

commit 15be3f5f130e81b31b6132ab2835ce7d4906cc18
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/425648
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=94a6a10423067051ff3d509774a66aa0ce7303f8
Submitter: Jenkins
Branch: stable/mitaka

commit 94a6a10423067051ff3d509774a66aa0ce7303f8
Author: Michael Polenchuk <email address hidden>
Date: Wed Jan 25 12:27:00 2017 +0400

    Adjust scope of neutron agent notify

    Shift notify under if condition to meet service resource relationship.
    Also function has been modified to exclude creation
    files with :: symbols.

    Change-Id: I2cbf9c4b2c7a9796b7b7ab53733cc489b03e533b
    Related-Bug: #1613724
    Closes-bug: #1658952