Openvswitch throwing a table missing error?

Bug #1315495 reported by J. Kary on 2014-05-02
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Undecided
Unassigned

Bug Description

I am running IceHouse with CentOS 6.4 and I followed the redhat deployment guide for my install. It is a 3 node setup with controller, network and compute nodes. My issue is happening on the compute node.

My /var/log/neutron/openvswitch-agent.log file is continually throwing this error:

2014-05-02 15:11:44.195 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-05-02 15:11:44.277 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
2014-05-02 15:11:44.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
2014-05-02 15:11:44.358 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])
2014-05-02 15:11:44.796 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Port fd5bb883-a415-49ec-849a-07442e3c7f1c updated. Details: {u'admin_state_up': True, u'network_id': u'5e377240-4297-44c5-aec3-90e7db89b7a2', u'segmentation_id': 2, u'physical_network': None, u'device': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'port_id': u'fd5bb883-a415-49ec-849a-07442e3c7f1c', u'network_type': u'gre'}
2014-05-02 15:11:44.854 2003 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Error while processing VIF ports
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1226, in rpc_loop
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent sync = self.process_network_ports(port_info)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1080, in process_network_ports
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices_added_updated)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 985, in treat_devices_added_or_updated
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.context, device, self.agent_id, cfg.CONF.host)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/agent/rpc.py", line 107, in update_device_up
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent topic=self.topic)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/proxy.py", line 125, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent result = rpc.call(context, real_topic, msg, timeout)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/__init__.py", line 112, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return _get_impl().call(CONF, context, topic, msg, timeout)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/impl_qpid.py", line 784, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent rpc_amqp.get_connection_pool(conf, Connection))
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 575, in call
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent rv = list(rv)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 540, in __iter__
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise result
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: OperationalError (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \nFROM ml2_port_bindings \nWHERE ml2_port_bindings.port_id LIKE ? || '%%'" (u'fd5bb883-a415-49ec-849a-07442e3c7f1c',)
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/amqp.py", line 462, in _process_data\n **args)\n', u' File "/usr/lib/python2.6/site-packages/neutron/common/rpc.py", line 45, in dispatch\n neutron_ctxt, version, method, namespace, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/rpc.py", line 196, in update_device_up\n if (host and not plugin.port_bound_to_host(port_id, host)):\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/plugin.py", line 782, in port_bound_to_host\n port_host = db.get_port_binding_host(port_id)\n', u' File "/usr/lib/python2.6/site-packages/neutron/plugins/ml2/db.py", line 130, in get_port_binding_host\n filter(models.PortBinding.port_id.startswith(port_id)).\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2184, in one\n ret = list(self)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in __iter__\n return self._execute_and_instances(context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances\n result = conn.execute(querycontext.statement, self._params)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute\n params)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement\n compiled_sql, distilled_params\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context\n context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context\n context)\n', u' File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute\n cursor.execute(statement, parameters)\n', u'OperationalError: (OperationalError) no such table: ml2_port_bindings u"SELECT ml2_port_bindings.port_id AS ml2_port_bindings_port_id, ml2_port_bindings.host AS ml2_port_bindings_host, ml2_port_bindings.vnic_type AS ml2_port_bindings_vnic_type, ml2_port_bindings.profile AS ml2_port_bindings_profile, ml2_port_bindings.vif_type AS ml2_port_bindings_vif_type, ml2_port_bindings.vif_details AS ml2_port_bindings_vif_details, ml2_port_bindings.driver AS ml2_port_bindings_driver, ml2_port_bindings.segment AS ml2_port_bindings_segment \\nFROM ml2_port_bindings \\nWHERE ml2_port_bindings.port_id LIKE ? || \'%%\'" (u\'fd5bb883-a415-49ec-849a-07442e3c7f1c\',)\n'].
2014-05-02 15:11:44.854 2003 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent
2014-05-02 15:11:46.196 2003 INFO neutron.plugins.openvswitch.agent.ovs_neutron_agent [-] Agent out of sync with plugin!
2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo93d39ce1-92', [u'map', [[u'attached-mac', u'fa:16:3e:37:26:f9'], [u'iface-id', u'93d39ce1-9209-49ee-96b5-01ff479b71c5'], [u'iface-status', u'active'], [u'vm-uuid', u'74187c71-8fed-4214-abb2-42c301489deb']]], -1]
2014-05-02 15:11:46.278 2003 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvo02722581-14', [u'map', [[u'attached-mac', u'fa:16:3e:1f:59:19'], [u'iface-id', u'02722581-14f4-4e6f-bfd2-8fc6594440da'], [u'iface-status', u'active'], [u'vm-uuid', u'3402975f-6a90-45d9-86e4-0f4ed00b2762']]], -1]
2014-05-02 15:11:46.359 2003 INFO neutron.agent.securitygroups_rpc [-] Preparing filters for devices set([u'831403fc-8d48-4949-8a4f-4c4f3455d81a', u'fd5bb883-a415-49ec-849a-07442e3c7f1c'])

however my ml2_port_bindings table does exist and has data:

mysql> desc ml2_port_bindings
    -> ;
+-------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------+------+-----+---------+-------+
| port_id | varchar(36) | NO | PRI | NULL | |
| host | varchar(255) | NO | | NULL | |
| vnic_type | varchar(64) | NO | | NULL | |
| profile | varchar(4095) | NO | | NULL | |
| vif_type | varchar(64) | NO | | NULL | |
| vif_details | varchar(4095) | NO | | NULL | |
| driver | varchar(64) | YES | | NULL | |
| segment | varchar(36) | YES | MUL | NULL | |
+-------------+---------------+------+-----+---------+-------+
8 rows in set (0.00 sec)

mysql> select * from ml2_port_bindings;
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
| port_id | host | vnic_type | profile | vif_type | vif_details | driver | segment |
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
| 1c525688-451c-457d-bfc4-b65ebcffcc5a | | normal | {} | unbound | | NULL | NULL |
| 27c6fae0-f7a4-4a7d-a39f-c6b8946d2cf2 | nmtg-net001.cisco.com | normal | {} | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 4f2d43cb-35f9-48f7-a715-6217aebf49fd | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 831403fc-8d48-4949-8a4f-4c4f3455d81a | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
| 8e24653d-1e5f-4fa1-91a4-c85434c4659b | nmtg-net001.cisco.com | normal | {} | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | 9321dddb-75fb-496b-bf9f-ecf85ee202a2 |
| fd5bb883-a415-49ec-849a-07442e3c7f1c | nmtg-cpu001.cisco.com | normal | | ovs | {"port_filter": true, "ovs_hybrid_plug": true} | openvswitch | a661a94f-0b70-4efe-b072-9dd3b25281fa |
+--------------------------------------+-----------------------+-----------+---------+----------+------------------------------------------------+-------------+--------------------------------------+
6 rows in set (0.00 sec)

mysql>

The compute node is able to ping and connect to the neutron database manually on the command line.

Did you make sure all the relevant db migrations were executed for your deployment?

To check:

neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutprol/plugins/ml2/ml2.ini current

Changed in neutron:
status: New → Incomplete
J. Kary (jkary-98) wrote :

Hi,

This is a fresh install from using the manual found on the openstack site.

[root@nmtg-ctrl001 ~]# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutprol/plugins/ml2/ml2.ini current
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 166, in main
    CONF()
  File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1638, in __call__
    raise ConfigFilesNotFoundError(self._namespace.files_not_found)
oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files: /etc/neutprol/plugins/ml2/ml2.ini
[root@nmtg-ctrl001 ~]#

I think you wanted me to run
[root@nmtg-ctrl001 ~]# neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini current
No handlers could be found for logger "neutron.common.legacy"
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 169, in main
    CONF.command.func(config, CONF.command.name)
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/cli.py", line 60, in do_alembic_command
    getattr(alembic_command, cmd)(config, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/alembic/command.py", line 189, in current
    script.run_env()
  File "/usr/lib/python2.6/site-packages/alembic/script.py", line 191, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.6/site-packages/alembic/util.py", line 186, in load_python_file
    module = imp.load_source(module_id, path, open(path, 'rb'))
  File "/usr/lib/python2.6/site-packages/neutron/db/migration/alembic_migrations/env.py", line 42, in <module>
    importutils.import_class(class_path)
  File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 33, in import_class
    traceback.format_exception(*sys.exc_info())))
ImportError: Class ml2 cannot be found (['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/neutron/openstack/common/importutils.py", line 28, in import_class\n __import__(mod_str)\n', 'ValueError: Empty module name\n'])
[root@nmtg-ctrl001 ~]#

Does this help?

Jason

Nicolae Paladi (n-paladi) wrote :

can confirm same bug, also CentOS 6.4

Nicolae Paladi (n-paladi) wrote :

I have the same issue, the deployment was installed with packstack on CentOS 6.4.
Can provide more info if needed.

One clue that I am investigating right now:
I get messages of the type
2014-06-18 18:55:53.323 7306 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'int-br-em2.401', [u'map', []], -1]
2014-06-18 18:55:53.324 7306 WARNING neutron.agent.linux.ovs_lib [-] Found failed openvswitch port: [u'qvob07eb20b-f4', [u'map', [[u'attached-mac', u'fa:16:3e:d4:94:53'], [u'iface-id', u'b07eb20b-f4ae-4740-a9bb-5cb827033c63'], [u'iface-status', u'active'], [u'vm-uuid', u'4f02745d-0ed0-4688-b494-da743b8e07c6']]], -1]

on all hosts when launching VM instances, with one exception.
I do not get this message, and the instances obtain their IP address as expected on the host where the L3 agent connected to the router is running.

Nicolae Paladi (n-paladi) wrote :

Hi J.Kary, did you manage to make any progress on this issue?

Tux Ninja (tuxninja) wrote :

Hi Nicolae...

Changing

firewall_driver=True

TO

firewall_driver=neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

in /etc/neutron/plugin.ini

Then restarting openstack services, worked for me...

See:
https://github.com/enovance/puppet-openstack-cloud/issues/12

Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers