Got the following logs. The unimportant part was delete for readability. Also comments were added for every line.
# Create a gateway interface for the subnet.
2018-02-16 10:04:37.152 14476 DEBUG neutron.agent.linux.keepalived [req-f3c3106a-d72b-4eb4-b9ba-3ba6ba3b35fd - 0d042481da874d229f542e35ef7ac589 - - -] VIP [fe80::f816:3eff:fe8c:7ef8/64, qr-8c8b4c01-40, link] was added add_vip /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:206
# Assign an IP address to the interface
2018-02-16 10:04:37.153 14476 DEBUG neutron.agent.linux.keepalived [req-f3c3106a-d72b-4eb4-b9ba-3ba6ba3b35fd - 0d042481da874d229f542e35ef7ac589 - - -] VIP [192.168.0.1/24, qr-8c8b4c01-40, None] was added add_vip /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:206
# Update keepalived configuration
2018-02-16 10:04:40.970 14476 DEBUG neutron.agent.linux.keepalived [req-f3c3106a-d72b-4eb4-b9ba-3ba6ba3b35fd - 0d042481da874d229f542e35ef7ac589 - - -] Keepalived spawned with config /var/lib/neutron/ha_confs/a35f384b-549e-41c6-8076-2283be384e1b/keepalived.conf spawn /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:447
...
# Create a new gateway interface for the subnet.
2018-02-16 10:06:11.728 14476 DEBUG neutron.agent.linux.keepalived [req-7ae4a777-fa22-42e1-b9a4-726d75ff3620 - 0d042481da874d229f542e35ef7ac589 - - -] VIP [fe80::f816:3eff:fe1e:4778/64, qr-7dc17e0a-97, link] was added add_vip /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:206
# Tries to assign an address but this operation fails as the address (192.168.0.1/24) is already in use.
2018-02-16 10:06:11.729 14476 DEBUG neutron.agent.linux.keepalived [req-7ae4a777-fa22-42e1-b9a4-726d75ff3620 - 0d042481da874d229f542e35ef7ac589 - - -] VIP [192.168.0.1/24, qr-7dc17e0a-97, None] already present in [<neutron.agent.linux.keepalived.KeepalivedVipAddress object at 0x7f6d5750cd90>, <neutron.agent.linux.keepalived.KeepalivedVipAddress object at 0x7f6d571ce7d0>, <neutron.agent.linux.keepalived.KeepalivedVipAddress object at 0x7f6d5722dc50>, <neutron.agent.linux.keepalived.KeepalivedVipAddress object at 0x7f6d57235490>, <neutron.agent.linux.keepalived.KeepalivedVipAddress object at 0x7f6d5750cd10>] add_vip /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:208
# Removes the gateway interface (qr-8c8b4c01-40) that was create at the first line of this log.
2018-02-16 10:06:12.829 14476 DEBUG neutron.agent.linux.keepalived [req-7ae4a777-fa22-42e1-b9a4-726d75ff3620 - 0d042481da874d229f542e35ef7ac589 - - -] remove_vips_vroutes_by_interface: qr-8c8b4c01-40 remove_vips_vroutes_by_interface /usr/lib/python2.7/dist-packages/neutron/agent/linux/keepalived.py:211
So my guess is the following. The bug happens when neutron-l3-agent don't have time to remove an old network interface and add a new one with the same IP address. As result the agent creates the new gateway interface, tries to assign the IP. It cannot do that as the address still in use then the agent deletes the old interface and generates configuration for keepalived. This configuration doesn't contain the IP address for the new gateway interface. Therefore the new gateway interface doesn't get the address.
More debug. I've slightly modified a couple of functions in neutron/ agent/linux/ keepalived. py. Just for debug purpose.
def add_vip(self, ip_cidr, interface_name, scope): dress(ip_ cidr, interface_name, scope)
self.vips. append( vip)
LOG.debug( 'VIP %s was added', vip)
LOG.debug( 'VIP %s already present in %s', vip, self.vips)
vip = KeepalivedVipAd
if vip not in self.vips:
else:
def remove_ vips_vroutes_ by_interface( self, interface_name): debug(' remove_ vips_vroutes_ by_interface: %s', interface_name)
LOG.
self.vips = [vip for vip in self.vips
if vip.interface_name != interface_name]
self. virtual_ routes. remove_ routes_ on_interface( interface_ name)
def remove_ vip_by_ ip_address( self, ip_address): debug(' remove_ vip_by_ ip_address: %s', ip_address)
LOG.
self.vips = [vip for vip in self.vips
if vip.ip_address != ip_address]
The original code can be found here https:/ /github. com/openstack/ neutron/ blob/master/ neutron/ agent/l3/ ha_router. py#L213
Got the following logs. The unimportant part was delete for readability. Also comments were added for every line.
# Create a gateway interface for the subnet. agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [fe80:: f816:3eff: fe8c:7ef8/ 64, qr-8c8b4c01-40, link] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [192.168.0.1/24, qr-8c8b4c01-40, None] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] Keepalived spawned with config /var/lib/ neutron/ ha_confs/ a35f384b- 549e-41c6- 8076-2283be384e 1b/keepalived. conf spawn /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:447 agent.linux. keepalived [req-7ae4a777- fa22-42e1- b9a4-726d75ff36 20 - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [fe80:: f816:3eff: fe1e:4778/ 64, qr-7dc17e0a-97, link] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-7ae4a777- fa22-42e1- b9a4-726d75ff36 20 - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [192.168.0.1/24, qr-7dc17e0a-97, None] already present in [<neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d5750cd90>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d571ce7d0>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d5722dc50>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d57235490>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d5750cd10>] add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:208 agent.linux. keepalived [req-7ae4a777- fa22-42e1- b9a4-726d75ff36 20 - 0d042481da874d2 29f542e35ef7ac5 89 - - -] remove_ vips_vroutes_ by_interface: qr-8c8b4c01-40 remove_ vips_vroutes_ by_interface /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:211
2018-02-16 10:04:37.152 14476 DEBUG neutron.
# Assign an IP address to the interface
2018-02-16 10:04:37.153 14476 DEBUG neutron.
# Update keepalived configuration
2018-02-16 10:04:40.970 14476 DEBUG neutron.
...
# Create a new gateway interface for the subnet.
2018-02-16 10:06:11.728 14476 DEBUG neutron.
# Tries to assign an address but this operation fails as the address (192.168.0.1/24) is already in use.
2018-02-16 10:06:11.729 14476 DEBUG neutron.
# Removes the gateway interface (qr-8c8b4c01-40) that was create at the first line of this log.
2018-02-16 10:06:12.829 14476 DEBUG neutron.
So my guess is the following. The bug happens when neutron-l3-agent don't have time to remove an old network interface and add a new one with the same IP address. As result the agent creates the new gateway interface, tries to assign the IP. It cannot do that as the address still in use then the agent deletes the old interface and generates configuration for keepalived. This configuration doesn't contain the IP address for the new gateway interface. Therefore the new gateway interface doesn't get the address.
Hope this will help.