Openvswitch throwing a table missing error?

[J. Kary]

I am running IceHouse with CentOS 6.4 and I followed the redhat deployment guide for my install. It is a 3 node setup with controller, network and compute nodes. My issue is happening on the compute node.

My /var/log/neutron/openvswitch-agent.log file is continually throwing this error:

2014-05-02 15:11:44.195 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-05-02 15:11:44.277 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
2014-05-02 15:11:44.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
2014-05-02 15:11:44.358 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])
2014-05-02 15:11:44.796 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port fd5bb883-a415-49ec-849a-07442e3c7f1c updated. Details: {u'admin_state_up': True, u'network_id': u'5e377240-4297-44c5-aec3-90e7db89b7a2', u'segmentation_id': 2, u'physical_network': None, u'device': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'port_id': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'network_type': u'gre'}
2014-05-02 15:11:44.854 2003 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1226, in rpc_loop
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent sync = self.process_network_ports(port_info)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1080, in process_network_ports
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices_added_updated)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 985, in treat_devices_added_or_updated
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.context, device, self.agent_id, cfg.CONF.host)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/agent/rpc.py", line 107, in update_device_up
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent topic=self.topic)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 125, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent result = rpc.call(context, real_topic, msg, timeout)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 112, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return _get_impl().call(CONF, context, topic, msg, timeout)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 784, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent rpc_amqp.get_connection_pool(conf, Connection))
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 575, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent rv = list(rv)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 540, in __iter__
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise result
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: OperationalError (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \nFROM ml2_port_bindings \nWHERE ml2_port_bindings.port_id LIKE ? || '%%'" (u'fd5bb883-a415-49ec-849a-07442e3c7f1c',)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 462, in _process_data\n **args)\n', u' File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n neutron_ctxt, version, method, namespace, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 196, in update_device_up\n if (host and not plugin.port_bound_to_host(port_id, host)):\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 782, in port_bound_to_host\n port_host = db.get_port_binding_host(port_id)\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/db.py", line 130, in get_port_binding_host\n filter(models.PortBinding.port_id.startswith(port_id)).\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one\n ret = list(self)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n params)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n cursor.execute(statement, parameters)\n', u'OperationalError: (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \\nFROM ml2_port_bindings \\nWHERE ml2_port_bindings.port_id LIKE ? || \'%%\'" (u\'fd5bb883-a415-49ec-849a-07442e3c7f1c\',)\n'].
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
2014-05-02 15:11:46.196 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
2014-05-02 15:11:46.359 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])

however my ml2_port_bindings table does exist and has data:

mysql> desc ml2_port_bindings
    -> ;
+-------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| port_id | varchar(36) | NO | PRI | NULL | |
| host | varchar(255) | NO | | NULL | |
| vnic_type | varchar(64) | NO | | NULL | |
| profile | varchar(4095) | NO | | NULL | |
| vif_type | varchar(64) | NO | | NULL | |
| vif_details | varchar(4095) | NO | | NULL | |
| driver | varchar(64) | YES | | NULL | |
| segment | varchar(36) | YES | MUL | NULL | |
+-------------+---------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from ml2_port_bindings;
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
| port_id | host | vnic_type | profile | vif_type | vif_details | driver | segment |
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
| 1c525688-451c-457d-bfc4-b65ebcffcc5a | | normal | {} | unbound | | NULL | NULL |
| 27c6fae0-f7a4-4a7d-a39f-c6b8946d2cf2 | nmtg-net001.cisco.com | normal | {} | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 4f2d43cb-35f9-48f7-a715-6217aebf49fd | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 831403fc-8d48-4949-8a4f-4c4f3455d81a | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 8e24653d-1e5f-4fa1-91a4-c85434c4659b | nmtg-net001.cisco.com | normal | {} | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | 9321dddb-75fb-496b-bf9f-ecf85ee202a2 |
| fd5bb883-a415-49ec-849a-07442e3c7f1c | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
6 rows in set (0.00 sec)

mysql>

The compute node is able to ping and connect to the neutron database manually on the command line.

[Salvatore Orlando (salvatore-orlando)]

Did you make sure all the relevant db migrations were executed for your deployment?

To check:

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutprol/plugins/ml2/ml2.ini current

[J. Kary (jkary-98)]

Hi,

This is a fresh install from using the manual found on the openstack site.

[root@nmtg-ctrl001 ~]# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutprol/plugins/ml2/ml2.ini current
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 166, in main
    CONF()
  File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1638, in __call__
    raise ConfigFilesNotFoundError(self._namespace.files_not_found)
oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files: /etc/neutprol/plugins/ml2/ml2.ini
[root@nmtg-ctrl001 ~]#

I think you wanted me to run
[root@nmtg-ctrl001 ~]# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini current
No handlers could be found for logger "neutron.common.legacy"
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 169, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 60, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/alembic/command.py", line 189, in current
    script.run_env()
  File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file
    module = imp.load_source(module_id, path, open(path, 'rb'))
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 42, in <module>
    importutils.import_class(class_path)
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 33, in import_class
    traceback.format_exception(*sys.exc_info())))
ImportError: Class ml2 cannot be found (['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 28, in import_class\n __import__(mod_str)\n', 'ValueError: Empty module name\n'])
[root@nmtg-ctrl001 ~]#

Does this help?

Jason

[Nicolae Paladi (n-paladi)]

can confirm same bug, also CentOS 6.4

[Nicolae Paladi (n-paladi)]

I have the same issue, the deployment was installed with packstack on CentOS 6.4.
Can provide more info if needed.

One clue that I am investigating right now:
I get messages of the type
2014-06-18 18:55:53.323 7306 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'int-br-em2.401', [u'map', []], -1]
2014-06-18 18:55:53.324 7306 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvob07eb20b-f4', [u'map', [[u'attached-mac', u'fa:16:3e:d4:94:53'], [u'iface-id', u'b07eb20b-f4ae-4740-a9bb-5cb827033c63'], [u'iface-status', u'active'], [u'vm-uuid', u'4f02745d-0ed0-4688-b494-da743b8e07c6']]], -1]

on all hosts when launching VM instances, with one exception.
I do not get this message, and the instances obtain their IP address as expected on the host where the L3 agent connected to the router is running.

[Nicolae Paladi (n-paladi)]

Hi J.Kary, did you manage to make any progress on this issue?

[Tux Ninja (tuxninja)]

Hi Nicolae...

Changing

firewall_driver=True

TO

firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

in /etc/neutron/plugin.ini

Then restarting openstack services, worked for me...

See:
https://github.com/enovance/puppet-openstack-cloud/issues/12

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]