odl client 404 on router deletion

[Isaku Yamahata]

some tempest tests causes 404 on ODL neutron northbound. and journal syncer doesn't handle it well.

This happens on error recovery of router ceration, the router is deleted explicitly in exception clause.
In that case, router to be deleted doesn't exist yet depending on where exception happens.

http://logs.openstack.org/38/484638/7/check/gate-tempest-dsvm-networking-odl-carbon-snapshot-vpnservice-v2driver-nv/e19e290/logs/screen-q-svc.txt.gz?level=AUDIT#_Aug_02_08_32_03_218707

Aug 02 08:32:03.218707 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client Traceback (most recent call last):
Aug 02 08:32:03.218864 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
Aug 02 08:32:03.219015 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client response.raise_for_status()
Aug 02 08:32:03.219163 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 937, in raise_for_status
Aug 02 08:32:03.219321 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client raise HTTPError(http_error_msg, response=self)
Aug 02 08:32:03.219467 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client HTTPError: 404 Client Error: Not Found for url: http://158.69.75.190:8087/controller/nb/v2/neutron/routers/12213a14-5174-4067-8eb3-8142a446cf16
Aug 02 08:32:03.219623 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client
Aug 02 08:32:03.219785 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.common.client [None req-08a0df76-0b20-4569-b6e2-c65bfa0ba1b2 None None] REST request ( delete ) to url ( routers/12213a14-5174-4067-8eb3-8142a446cf16 ) is failed. Request body : [None] service: HTTPError: 404 Client Error: Not Found for url: http://158.69.75.190:8087/controller/nb/v2/neutron/routers/12213a14-5174-4067-8eb3-8142a446cf16
Aug 02 08:32:03.220891 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal [None req-08a0df76-0b20-4569-b6e2-c65bfa0ba1b2 None None] Error while processing delete router 12213a14-5174-4067-8eb3-8142a446cf16: HTTPError: 404 Client Error: Not Found for url: http://158.69.75.190:8087/controller/nb/v2/neutron/routers/12213a14-5174-4067-8eb3-8142a446cf16
Aug 02 08:32:03.221141 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal Traceback (most recent call last):
Aug 02 08:32:03.221382 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/journal/journal.py", line 245, in _sync_entry
Aug 02 08:32:03.221592 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal self.client.sendjson(method, urlpath, to_send)
Aug 02 08:32:03.221823 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 106, in sendjson
Aug 02 08:32:03.222090 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal 'body': obj})
Aug 02 08:32:03.222375 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 02 08:32:03.222632 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal self.force_reraise()
Aug 02 08:32:03.222886 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 02 08:32:03.223126 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
Aug 02 08:32:03.223381 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 98, in sendjson
Aug 02 08:32:03.223627 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal self.request(method, urlpath, data))
Aug 02 08:32:03.223876 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 140, in _check_response
Aug 02 08:32:03.224125 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal {'e': error, 'text': response.text}, exc_info=1)
Aug 02 08:32:03.224406 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 02 08:32:03.224675 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal self.force_reraise()
Aug 02 08:32:03.224920 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 02 08:32:03.225166 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal six.reraise(self.type_, self.value, self.tb)
Aug 02 08:32:03.225435 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/opt/stack/new/networking-odl/networking_odl/common/client.py", line 136, in _check_response
Aug 02 08:32:03.225684 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal response.raise_for_status()
Aug 02 08:32:03.225930 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 937, in raise_for_status
Aug 02 08:32:03.226348 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal raise HTTPError(http_error_msg, response=self)
Aug 02 08:32:03.226656 ubuntu-xenial-ovh-bhs1-10209073 neutron-server[24896]: ERROR networking_odl.journal.journal HTTPError: 404 Client Error: Not Found for url: http://158.69.75.190:8087/controller/nb/v2/neutron/routers/12213a14-5174-4067-8eb3-8142a446cf16

[Isaku Yamahata (yamahata)]

This may caused by l3 plugin bug. so possibly l3 flavor may solve this.
So for now this is low priority and assigned to me.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/331382
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.