fs017: failure to delete IP causing tempest failures

[Matt Young]

master promotion is being blocked by a failure in periodic-multinode-1ctlr-featureset017

master: 66b3734b4a57941c2d66dfc7d7961b2925c46b92_e4e594a6

---

http://38.145.34.55/master.log

2018-02-28 20:21:26,221 2381 INFO promoter Skipping promotion of tripleo-ci-testing to current-tripleo, missing successful jobs: ['periodic-multinode-1ctlr-featureset016', 'periodic-multinode-1ctlr-featureset017', 'periodic-multinode-1ctlr-featureset019']

---

It appears that tempest is failing telemetry tests:

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/tempest.html.gz

(hypothesis from mwhahaha) a failure to delete an IP, or failing to delete it fast enough is causing this

- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-02-28_19_13_39_116

---

2018-02-28 19:13:38.980 27 INFO neutron.wsgi [req-8500f497-a51a-45b6-9713-5d4fd8bcb500 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] 192.168.24.3 "DELETE /v2.0/routers/c8260734-ffe9-4450-a3c7-a4cff0fdc42e HTTP/1.1" status: 204 len: 168 time: 1.3387601
2018-02-28 19:13:38.982 27 DEBUG neutron.wsgi [-] (27) accepted ('192.168.24.3', 53010) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:883
2018-02-28 19:13:39.041 27 DEBUG neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Deleting subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5 delete_subnet /usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py:999
2018-02-28 19:13:39.042 27 DEBUG neutron_lib.callbacks.manager [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Notify callbacks [] for subnet, before_delete _notify_loop /usr/lib/python2.7/site-packages/neutron_lib/callbacks/manager.py:167
2018-02-28 19:13:39.116 27 INFO neutron.db.db_base_plugin_v2 [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] Found port (48798d40-cdc3-415a-9244-2dfaac84f134, 10.100.0.6) having IP allocation on subnet 87f0f9b4-edab-454d-a7d9-0144816b54b5, cannot delete
2018-02-28 19:13:39.118 27 INFO neutron.pecan_wsgi.hooks.translation [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] DELETE failed (client error): There was a conflict when trying to complete your request.
2018-02-28 19:13:39.118 27 DEBUG neutron.pecan_wsgi.hooks.notifier [req-eef89b52-7b20-497b-af68-f575151fed76 c090e652604f44a08d3498250ad90595 f0f25c526c484052ab6b4af65d114e51 - default default] No notification will be sent due to unsuccessful status code: 409 after /usr/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/notifier.py:79

[Matt Young (halcyondude)]

Reproducer script (run after sourcing RDO Cloud creds)

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/reproducer-quickstart.sh

[wes hayutin (weshayutin)]

OK.. I have a recreate up, Chandan and Arx have seen this issue before. It appears to be a bug in tempest initially, so we'll drive it with the tempest guys

[Matt Young (halcyondude)]

I have a second recreate as well if it's needed

[Mehdi Abaakouk (sileht)]

Issue the stack does not scale up:

Gnocchi metricd does not have measues to compute (only "0 metrics processed" msg):

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/subnode-2/var/log/containers/gnocchi/gnocchi-metricd.log.txt.gz

Gnocchi API never receive any measures from Ceilometer:

(No "POST /v1/resource/<type>/<id>/metric/<metric_name>/measures" call)

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/subnode-2/var/log/containers/httpd/gnocchi-api/gnocchi_wsgi_access.log.txt.gz

Ceilometer seems to built all needed metrics, but it doesn't sent them to Gnocchi:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/subnode-2/var/log/containers/ceilometer/

Ceilometer fail to create/check archive policy configuration:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/45442f0/subnode-2/var/log/containers/ceilometer/agent-notification.log.txt.gz#_2018-02-28_18_52_10_888

This is related to a recent change: https://review.openstack.org/#/c/530885/

Maybe Ceilometer should retry in this case.

[Matt Young (halcyondude)]

https://review.openstack.org/#/c/548917 was posted to ceilometer that should address this

[wes hayutin (weshayutin)]

this has been fixed. thanks!

[wes hayutin (weshayutin)]

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/e3c8e77/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 11.0.0 release.