neutron fails to connect to q-agent-notifier-port-delete_fanout exchange

[Pavlo Shchelokovskyy]

logstash query (>70 hits for last 7 days)
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22NotFound%3A%20Basic.publish%3A%20(404)%20NOT_FOUND%20-%20no%20exchange%20'q-agent-notifier-port-'%5C%22%20AND%20build_status%3AFAILURE

example of the failed log
http://logs.openstack.org/27/469627/8/check/gate-grenade-dsvm-ironic-ubuntu-xenial/bffe275/logs/new/screen-q-svc.txt.gz#_2017-06-08_08_23_28_906

after 3 attempts neutron seems to give up.
The actual tempest tests failing are seemingly random (most probably whichever tries to cleanup a port created during test), failing on 500 from Neutron.

interestingly enough, this mostly affects single-node standard grenade jobs for ironic and ironic-inspector (which also installs ironic).

[Pavlo Shchelokovskyy (pshchelo)]

I also see similar failures for port-update_fanout

http://logs.openstack.org/27/469627/9/check/gate-grenade-dsvm-ironic-ubuntu-xenial/a346d4c/logs/new/screen-q-svc.txt.gz#_2017-06-08_12_21_37_848

[milan k (vetrisko)]

Ah so inspector was already mentioned, maybe I needn't also-affect it

[Kevin Benton (kevinbenton)]

I wonder why the exchange isn't being declared. I wonder if we are hitting an oslo messaging bug here of some sort.

[Pavlo Shchelokovskyy (pshchelo)]

E-R query is proposed at https://review.openstack.org/479237

[Vladyslav Drok (vdrok)]

This becomes kind of a blocker for our grenade job.

[Anup (anup-d-navare)]

This bug seem to be making Ironic grenade job fail in many number of patches.