duplicate port create requests to neutron by compute

[ashwin raveendran]

When quantum server is slow in responding to the port create requests (30 seconds or more) we are seeing duplicate port requests being made for the same operation. This is related to the value set for the quantum_url_timeout parameter (30 seconds by default) and by increasing the value we get duplicate port requests only for those quantum requests taking longer than the new timeout value.
It appears that the quantum client is retrying the request on timeouts and is an error to do it for POST which is non-idempotent and in this case creates duplicate ports. Further investigation of the problem shows that this is coming from the httplib2 library which retries requests on socket time out. https://code.google.com/p/httplib2/issues/detail?id=109

[Sudheendra Murthy (sudhi-vm)]

There is a retry logic in httplib2 used by neutron that automatically retries requests. One instance when this kicks in is: client makes a connection —> issues request to the server —> server takes too long to respond (if busy) —> client socket timesout waiting for response and client goes for a retry. The end result is that the request may get processed twice at the server. There is a parameter RETRIES used in _conn_request(…) defined in __init__.py. This parameter is not configurable, hard-coded to 2.

One possible way to deal with this is to increase the timeout value (which is configurable) to a larger value. The default is 30 seconds and increasing it to larger value helps. This will not solve the issue, but could make it less likely to occur.

Here are Google references to the retry logic in httplib2:
https://code.google.com/p/httplib2/issues/detail?id=109
https://groups.google.com/forum/#!msg/google-api-python-client/VE1d7a7Wevg/elmt1MDsjmAJ
http://httplib2.googlecode.com/hg/doc/html/libhttplib2.html

[Ravi Chunduru (ravivsn)]

There are no retries in case of PUT or POST requests in neutron client.
https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1239

Retries are only for GET, DELETE and LIST only
https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1235

Retry logic is here
https://github.com/openstack/python-neutronclient/blob/master/neutronclient/v2_0/client.py#L1211

The bug is in httplib2 implementation. I see the fix is also available
https://code.google.com/p/httplib2/source/detail?r=4adde1593f41#

[Ravi Chunduru (ravivsn)]

In addition to it, I agree we need to set httplib2 RETRIES to 1 and leave the retry logic with neutron client.

[Eugene Nikanorov (enikanorov)]

Right now neutron client uses requests library, not httplib2, so it may only apply to old client.

[Rossella Sblendido (rossella-o)]

This was fixed by https://review.openstack.org/#/c/65840/ no idea why this bug's status is not showing it