Per discussion with Joseph, this issue was encountered on HW lab during storage regression testcase storage/test_filesystem.py::test_controllerfs_mod_when_host_locked. While the testcase itself is reported as passed, the neutron-ovs-agent log begins reporting failure to report state: {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [-] Failed reporting state!: MessagingTimeout: Timed out waiting for a reply to message ID af2b0df78e5b441db4a151c982906da4\n","stream":"stdout","time":"2019-08-18T15:47:13.270851422Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):\n","stream":"stdout","time":"2019-08-18T15:47:13.270901208Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py\", line 346, in _report_state\n","stream":"stdout","time":"2019-08-18T15:47:13.270909837Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent True)\n","stream":"stdout","time":"2019-08-18T15:47:13.27092587Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/neutron/agent/rpc.py\", line 101, in report_state\n","stream":"stdout","time":"2019-08-18T15:47:13.270932796Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return method(context, 'report_state', **kwargs)\n","stream":"stdout","time":"2019-08-18T15:47:13.27093909Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 178, in call\n","stream":"stdout","time":"2019-08-18T15:47:13.270954099Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)\n","stream":"stdout","time":"2019-08-18T15:47:13.270960567Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 128, in _send\n","stream":"stdout","time":"2019-08-18T15:47:13.270966461Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)\n","stream":"stdout","time":"2019-08-18T15:47:13.270972536Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 645, in send\n","stream":"stdout","time":"2019-08-18T15:47:13.270978171Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout, retry=retry)\n","stream":"stdout","time":"2019-08-18T15:47:13.270984472Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 634, in _send\n","stream":"stdout","time":"2019-08-18T15:47:13.270991265Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent call_monitor_timeout)\n","stream":"stdout","time":"2019-08-18T15:47:13.270997652Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 520, in wait\n","stream":"stdout","time":"2019-08-18T15:47:13.271003869Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout)\n","stream":"stdout","time":"2019-08-18T15:47:13.271010341Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File \"/var/lib/openstack/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 397, in get\n","stream":"stdout","time":"2019-08-18T15:47:13.271023313Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id)\n","stream":"stdout","time":"2019-08-18T15:47:13.271030369Z"} {"log":"2019-08-18 15:47:13.263 98 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID af2b0df78e5b441db4a151c982906da4\n","stream":"stdout","time":"2019-08-18T15:47:13.271036148Z"} PASS 20190818 15:45:32 testcases/functional/storage/test_filesystem.py::test_controllerfs_mod_when_host_locked The sequence of commands run from controller-0 were as follows: controller-1 host locked 2019-08-18T15:45:40.000 controller-0 system .... host-lock controller-1 …. 2019-08-18T15:46:03.000 controller-0 system ... controllerfs-show database system controllerfs-modify database 2019-08-18T15:46:05.000 controller-0 system....controllerfs-modify database="21" unlock controller-1 2019-08-18T15:46:18.000 controller-0 system .... host-unlock controller-1 $ source /etc/platform/openrc ; system show +----------------------+------------------------------------------------------+ | Property | Value | +----------------------+------------------------------------------------------+ | contact | None | | created_at | 2019-08-17T19:02:57.185203+00:00 | | description | yow-cgcs-ironpass-20_27: setup by deployment manager | | https_enabled | False | | location | None | | name | yow-cgcs-ironpass-20-27 | | region_name | RegionOne | | sdn_enabled | False | | security_feature | spectre_meltdown_v1 | | service_project_name | services | | software_version | 19.01 | | system_mode | duplex | | system_type | Standard | | timezone | UTC | | updated_at | 2019-08-17T19:12:07.430304+00:00 | | uuid | f3ad3520-2d0c-42c5-9283-1da41338087c | | vswitch_type | none