Activity log for bug #1385234

Date Who What changed Old value New value Message
2014-10-24 11:32:49 Giulio Fidente bug added bug
2014-10-24 11:33:32 Giulio Fidente bug task added oslo.messaging
2014-10-24 11:33:44 Giulio Fidente bug task added neutron
2014-10-24 11:34:09 Giulio Fidente summary OVS tunneling between multiple neutron nodes breaks if amqp is restarted OVS tunneling between multiple neutron nodes misconfigured if amqp is restarted
2014-10-24 11:38:03 Giulio Fidente description At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ deployment due to rabbit cluster configuration. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging
2014-10-24 11:45:51 Giulio Fidente description At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them or to the computes. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging
2014-10-24 11:46:34 Giulio Fidente description At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them or to the computes. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them or to the computes. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ tripleo overcloud deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging
2014-10-24 12:38:01 Jakub Libosvar bug added subscriber Jakub Libosvar
2014-10-24 13:30:12 Giulio Fidente description At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them or to the computes. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node, or even _during_ tripleo overcloud deployment due to rabbit cluster configuration. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging At completion of a deployment with multiple controllers, by observing the gre tunnels created in OVS by the neutron ovs-agent, one will find that some neutron nodes may miss the tunnels in between them or to the computes. This is due to ovs-agents getting disconnected from the rabbit cluster without them noticing and as a result, being unable to receive updates from other nodes or publish updates. The disconnection may happen following a reconfig of a rabbit node, the VIP moving over a different node when rabbit is load balanced, or even _during_ tripleo overcloud deployment due to rabbit cluster configuration changes. This was observed using Kombu 3.0.33 as well as 2.5. Use of some aggressive (low) kernel keepalive probes interval seems to improve the reliability but a more appropriate fix seems to be support for heartbeat in oslo.messaging
2014-10-24 14:29:25 Giulio Fidente oslo.messaging: status New Incomplete
2014-10-24 14:29:29 Giulio Fidente oslo.messaging: status Incomplete New
2014-10-24 14:50:20 Eugene Nikanorov neutron: importance Undecided High
2014-10-30 14:53:43 Ben Nemec tripleo: status New Triaged
2014-11-10 01:47:43 Koji Iida bug added subscriber Koji Iida
2014-11-16 20:06:58 Romil Gupta neutron: assignee Romil Gupta (romilg)
2014-11-21 10:24:20 Eugene Nikanorov tags ovs
2014-11-21 10:24:32 Eugene Nikanorov neutron: importance High Medium
2014-11-21 10:24:50 Eugene Nikanorov neutron: status New Confirmed
2014-12-03 13:25:30 Mehdi Abaakouk oslo.messaging: status New Incomplete
2014-12-05 12:25:05 Giulio Fidente bug added subscriber Jan Provaznik
2014-12-17 18:01:05 OpenStack Infra tripleo: status Triaged In Progress
2014-12-17 18:01:05 OpenStack Infra tripleo: assignee Giulio Fidente (gfidente)
2014-12-18 21:31:12 OpenStack Infra tripleo: status In Progress Fix Committed
2014-12-24 10:00:46 Derek Higgins tripleo: status Fix Committed Fix Released
2015-04-28 14:19:21 Tomoko Inoue bug added subscriber Tomoko Inoue
2016-03-12 01:27:55 Armando Migliaccio neutron: status Confirmed Incomplete
2016-03-12 01:27:55 Armando Migliaccio neutron: assignee Romil Gupta (romilg)
2018-12-04 22:23:26 Ben Nemec oslo.messaging: status Incomplete Fix Released
2022-11-30 09:28:07 Rodolfo Alonso neutron: status Incomplete Won't Fix