Incorrect file being edited to configure ML2 plugin on the network node

[Jeff Shantz]

I have run through the entirety of chapters 1 - 7 on CentOS 6.5 twice now. Both times, after trying to start the neutron-openvswitch-agent on the network node at the end of section "Configure network node" in Chapter 7, the service fails to start and I get errors in /var/log/neutron/openvswitch-agent.log:

======
2014-04-14 06:38:22.380 3996 CRITICAL neutron [req-39f1e4f7-3896-4b20-9e8d-95faafed275a None] 'NoneType' object has no attribute 'rpartition'
2014-04-14 06:38:22.380 3996 TRACE neutron Traceback (most recent call last):
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/bin/neutron-openvswitch-agent", line 10, in <module>
2014-04-14 06:38:22.380 3996 TRACE neutron sys.exit(main())
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1359, in main
2014-04-14 06:38:22.380 3996 TRACE neutron agent = OVSNeutronAgent(**agent_config)
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 222, in __init__
2014-04-14 06:38:22.380 3996 TRACE neutron root_helper)
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 113, in __init__
2014-04-14 06:38:22.380 3996 TRACE neutron self.init_firewall(defer_refresh_firewall=True)
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py", line 140, in init_firewall
2014-04-14 06:38:22.380 3996 TRACE neutron self.firewall = importutils.import_object(firewall_driver)
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 39, in import_object
2014-04-14 06:38:22.380 3996 TRACE neutron return import_class(import_str)(*args, **kwargs)
2014-04-14 06:38:22.380 3996 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 27, in import_class
2014-04-14 06:38:22.380 3996 TRACE neutron mod_str, _sep, class_str = import_str.rpartition('.')
2014-04-14 06:38:22.380 3996 TRACE neutron AttributeError: 'NoneType' object has no attribute 'rpartition'
2014-04-14 06:38:22.380 3996 TRACE neutron
======

After debugging the problem and adding some print statements to the Python files in the stack trace, I determined that the firewall_driver variable in /usr/lib/python2.6/site-packages/neutron/agent/securitygroups_rpc.py (line 140) was set to None.

This was strange, since it is configured in /etc/neutron/plugins/ml2/ml2_conf.ini, as instructed in the documentation:

======
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
======

Taking a look at /etc/init.d/neutron-openvswitch-agent, we see that the following configuration files are used to start the service:

======
prog=$proj-$plugin
exec="/usr/bin/$prog"
configs=(
    "/usr/share/$proj/$proj-dist.conf" \
    "/etc/$proj/$proj.conf" \
    "/etc/$proj/plugins/openvswitch/ovs_neutron_plugin.ini" \
)
======

Nowhere is ml2_conf.ini (or the symlinked plugin.ini) referenced. If we try starting the service manually with the proper configuration files, it starts just fine:

======
/usr/bin/neutron-openvswitch-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini
======

So, it would appear that either the init script is out of date (indicating a dev bug), or the docs are out of date. If it's the latter, there are two options:

1. Delete /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini and link it to /etc/neutron/plugins/ml2/ml2_conf.ini:

   rm /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
   ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini

2. Edit /etc/init.d/neutron-openvswitch-agent to reference /etc/neutron/plugin.ini instead of ovs_neutron_plugin.ini:

  prog=$proj-$plugin
  exec="/usr/bin/$prog"
  configs=(
      "/usr/share/$proj/$proj-dist.conf" \
      "/etc/$proj/$proj.conf" \
      "/etc/$proj/plugin.ini" \
  )

I have confirmed that both solutions work just fine and the service starts properly after making either of these changes.

-----------------------------------
Built: 2014-04-14T05:54:22 00:00
git SHA: 69e9a932b042f09d26067dbca287e4bfb1fdbf11
URL: http://docs.openstack.org/trunk/install-guide/install/yum/content/neutron-ml2-network-node.html
source File: file:/home/jenkins/workspace/openstack-install-deploy-guide-fedora/doc/install-guide/section_neutron-ml2-network-node.xml
xml:id: neutron-ml2-network-node

[Jeff Shantz (jeffshantz)]

Same problem exists when starting the neutron-openvswitch-agent service on the compute node: http://docs.openstack.org/trunk/install-guide/install/yum/content/neutron-ml2-compute-node.html

I assume this is just a packaging bug, and not actually a doc bug?

[Matt Kassawara (ionosphere80)]

This is a packaging bug. RDO packages the Open vSwitch plug-in and agent together and the agent assumes that the environment uses the Open vSwitch plug-in. On the other hand, the package containing neutron-server looks for the plugin.ini symlink to determine the plug-in. The Open vSwitch agent should use a similar method. Given the upcoming release in several days, I think we should develop a temporary workaround for distributions using RDO packages. A couple of options come to mind:

1) Configure the appropriate keys in ovs_plugin_conf.ini.
2) Create a symlink from ovs_plugin_conf.ini to ml2_conf.ini.
3) Modify the init script to look for plugin.ini.

# diff neutron-openvswitch-agent neutron-openvswitch-agent.orig
18c18
< "/etc/$proj/plugin.ini" \
---
> "/etc/$proj/plugins/openvswitch/ovs_neutron_plugin.ini" \

We can probably modify the init script using sed to make it easy.

[Matt Kassawara (ionosphere80)]

Correction...

ovs_plugin_conf.ini -> ovs_neutron_plugin.ini

[Matt Kassawara (ionosphere80)]

I opened the following bug with RDO:

https://bugzilla.redhat.com/show_bug.cgi?id=1087647

[Jeff Shantz (jeffshantz)]

Agreed that sed is a good option.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/87415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/87415
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=870f6fd2814af07f220c4ecb66ae3430eb6ca2dd
Submitter: Jenkins
Branch: master

commit 870f6fd2814af07f220c4ecb66ae3430eb6ca2dd
Author: Matt Kassawara <email address hidden>
Date: Mon Apr 14 18:04:26 2014 -0600

    Created workaround for RDO packaging bug with Neutron

    I created a temporary workaround for a RDO packaging bug with
    Neutron that affects environments using the ML2 plug-in.

    https://bugzilla.redhat.com/show_bug.cgi?id=1087647

    Change-Id: Ief2831fea269828fb4a6c14e767de3eb66816211
    Closes-Bug: #1307326

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.