Relevant logs @ http://logs.openstack.org/38/496638/7/check/gate-rally-dsvm-networking-odl-carbon-snapshot/d1f32b9/logs/
q-svc log
=========
INFO networking_odl.journal.journal [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Processing - update port 5cd5a60a-06a2-4274-a7b0-b88ed937a49f
DEBUG networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Sending METHOD (put) URL (http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f) JSON ({
"port": {
"allowed_address_pairs": [],
"extra_dhcp_opts": [],
"updated_at": "2017-08-27T11:58:14Z",
"device_owner": "network:dhcp",
"revision_number": 3,
"port_security_enabled": false,
"binding:profile": {},
"fixed_ips": [
{
"subnet_id": "e272c54e-43be-47aa-acd0-c2aaab596862",
"ip_address": "10.2.15.2"
}
],
"security_groups": [],
"binding:vif_details": {
"support_vhost_user": false,
"has_datapath_type_netdev": false,
"uuid": "9aafa204-dacc-4a42-80c6-a3a05de43393",
"host_addresses": [
"ubuntu-xenial-inap-mtl01-10650239"
]
},
"binding:vif_type": "ovs",
"mac_address": "fa:16:3e:05:4e:41",
"binding:host_id": "ubuntu-xenial-inap-mtl01-10650239",
"description": "",
"tags": [],
"device_id": "dhcp8981c717-8b49-521d-af27-33139dcc5f8e-f3d90580-4bef-4d7c-b32b-af4576f42e67",
"name": "",
"admin_state_up": true,
"created_at": "2017-08-27T11:58:06Z",
"binding:vnic_type": "normal"
}
}) {{(pid=20895) request /opt/stack/new/networking-odl/networking_odl/common/client.py:89}}
DEBUG neutron.scheduler.dhcp_agent_scheduler [None req-b815a515-fef2-4e46-8dce-99ab43cf0dd5 c_rally_f2ccc531_1M2EcU8x c_rally_f2ccc531_8fzScwr5] Network 0414e07d-aaf2-4abc-843b-3da23b8eefa1 is scheduled to be hosted by DHCP agent 30f28597-4e0a-4350-833e-7df8e1aec4e0 {{(pid=20892) bind /opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py:199}}
DEBUG neutron.scheduler.dhcp_agent_scheduler [None req-fc80da8c-ad42-4dee-aa0e-aad6636f2f12 c_rally_f2ccc531_1M2EcU8x c_rally_f2ccc531_8fzScwr5] Network 39a52e67-9195-44ff-8d67-4d75c85fa703 is scheduled to be hosted by DHCP agent 30f28597-4e0a-4350-833e-7df8e1aec4e0 {{(pid=20892) bind /opt/stack/new/neutron/neutron/scheduler/dhcp_agent_scheduler.py:199}}
DEBUG networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Exception from ODL: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f Port UUID does not exist. {{(pid=20895) _check_response /opt/stack/new/networking-odl/networking_odl/common/client.py:140}}
ERROR networking_odl.common.client Traceback (most recent call last):
ERROR networking_odl.common.client File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
ERROR networking_odl.common.client response.raise_for_status()
ERROR networking_odl.common.client File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 935, in raise_for_status
ERROR networking_odl.common.client raise HTTPError(http_error_msg, response=self)
ERROR networking_odl.common.client HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
ERROR networking_odl.common.client
ERROR networking_odl.common.client [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] REST request ( put ) to url ( ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f ) is failed. Request body : [{u'port': {'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'updated_at': '2017-08-27T11:58:14Z', 'device_owner': u'network:dhcp', 'revision_number': 3, 'port_security_enabled': False, 'binding:profile': {}, 'fixed_ips': [{'subnet_id': u'e272c54e-43be-47aa-acd0-c2aaab596862', 'ip_address': u'10.2.15.2'}], 'security_groups': [], 'binding:vif_details': {u'support_vhost_user': False, u'has_datapath_type_netdev': False, u'uuid': u'9aafa204-dacc-4a42-80c6-a3a05de43393', u'host_addresses': [u'ubuntu-xenial-inap-mtl01-10650239']}, 'binding:vif_type': u'ovs', 'mac_address': u'fa:16:3e:05:4e:41', 'binding:host_id': u'ubuntu-xenial-inap-mtl01-10650239', 'description': u'', 'tags': [], 'device_id': u'dhcp8981c717-8b49-521d-af27-33139dcc5f8e-f3d90580-4bef-4d7c-b32b-af4576f42e67', 'name': u'', 'admin_state_up': True, 'created_at': '2017-08-27T11:58:06Z', 'binding:vnic_type': u'normal'}}] service: HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
ERROR networking_odl.journal.journal [None req-897486ef-e733-4835-816f-f7dd22c90760 None None] Error while processing update port 5cd5a60a-06a2-4274-a7b0-b88ed937a49f: HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
ERROR networking_odl.journal.journal Traceback (most recent call last):
ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/journal/journal.py", line 245, in _sync_entry
ERROR networking_odl.journal.journal self.client.sendjson(method, urlpath, to_send)
ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 106, in sendjson
ERROR networking_odl.journal.journal 'body': obj})
ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR networking_odl.journal.journal self.force_reraise()
ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 98, in sendjson
ERROR networking_odl.journal.journal self.request(method, urlpath, data))
ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 140, in _check_response
ERROR networking_odl.journal.journal {'e': error, 'text': response.text}, exc_info=1)
ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR networking_odl.journal.journal self.force_reraise()
ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
ERROR networking_odl.journal.journal response.raise_for_status()
ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 935, in raise_for_status
ERROR networking_odl.journal.journal raise HTTPError(http_error_msg, response=self)
ERROR networking_odl.journal.journal HTTPError: 404 Client Error: Not Found for url: http://198.72.124.116:8087/controller/nb/v2/neutron/ports/5cd5a60a-06a2-4274-a7b0-b88ed937a49f
ERROR networking_odl.journal.journal
While indeed alerting, there isn't enough information in the logs to identify exactly what caused the bug (current suspicion is some problem in the dependency calculations, but without more info it's hard to know for sure).
I'll send a fix to add more information in the logging that will allow us to further understand such scenarios.