ironic CI regression: dnsmasq doesn't respond to dhcp request

[Vasyl Saienko]

All jobs that uses flat network_interface are failed because bootstrap can't get IP address from DHCP server.

An example of failed job is:
http://logs.openstack.org/38/447538/6/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/f57afee/logs/

In the syslog we can see that DHCP doesn't respond to requests:

http://logs.openstack.org/38/447538/6/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/f57afee/logs/syslog.txt.gz#_Apr_18_12_30_00

Apr 18 12:30:00ubuntu-xenial-internap-mtl01-8463102 dnsmasq-dhcp[3453]: DHCPDISCOVER(tap6a904c1b-03) 52:54:00:f3:12:ee no address available
Apr 18 12:30:00ubuntu-xenial-internap-mtl01-8463102 ntpd[1715]: Listen normally on 15 vnet0 [fe80::fc54:ff:fef3:12ee%19]:123
Apr 18 12:30:00ubuntu-xenial-internap-mtl01-8463102 ntpd[1715]: new interface(s) found: waking up resolver
Apr 18 12:30:01ubuntu-xenial-internap-mtl01-8463102 dnsmasq-dhcp[3453]: DHCPDISCOVER(tap6a904c1b-03) 52:54:00:f3:12:ee no address available
Apr 18 12:30:03ubuntu-xenial-internap-mtl01-8463102 dnsmasq-dhcp[3453]: DHCPDISCOVER(tap6a904c1b-03) 52:54:00:f3:12:ee no address available
Apr 18 12:30:07ubuntu-xenial-internap-mtl01-8463102 dnsmasq-dhcp[3453]: DHCPDISCOVER(tap6a904c1b-03) 52:54:00:f3:12:ee no address available

[Vasyl Saienko (vsaienko)]

patch that caused regression is: https://review.openstack.org/#/c/447538

[Vasyl Saienko (vsaienko)]

fix https://review.openstack.org/#/c/457904/

[Derek Higgins (derekh)]

Have you any info on the error this patch caused? or was the regression found with a bisect?

[Vladyslav Drok (vdrok)]

Derek, I think just by investigating the recently merged patches. The one being reverted has failed the ironic job with the same symptoms.

[Akihiro Motoki (amotoki)]

I confirmed https://review.openstack.org/#/c/447538 causes the regression that IPv4 address is lost when MAC address is updated. I agree to revert https://review.openstack.org/#/c/447538 and revisit bug 1671548 again.

After https://review.openstack.org/#/c/447538 is applied (the current neutron master branch), when I updated a MAC address of port with both IPv4 and IPv6 addresses, IPv4 address was lost and IPv6 address was updated.
http://paste.openstack.org/show/607198/

When I reverted https://review.openstack.org/#/c/447538, even when MAC address of a port is updated, IPv4 was not lost (while IPv6 address was not updated too -- this is a bug reported in bug 1671548 and it is not a problem here).
http://paste.openstack.org/show/607199/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/457904
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=37097a991b837b5508b4eb85649854a6c07cc280
Submitter: Jenkins
Branch: master

commit 37097a991b837b5508b4eb85649854a6c07cc280
Author: Vasyl Saienko <email address hidden>
Date: Wed Apr 19 07:47:09 2017 +0000

    Revert "Update auto-addresses on MAC change"

    Original patch caused ironic CI regression.

    This reverts commit 27746d1d16ceec59ca6576d565e5e4157427fa96.

    Closes-Bug: #1684038

    Change-Id: I29afcfac626f4947ad4db288185208c2c5c2b7a1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/458293

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/458343

[Vladyslav Drok (vdrok)]

I suppose ironic side of the bug can be closed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458343
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=df320474c5d07ecba0f0ac5cd73a6d5be66dc0a2
Submitter: Jenkins
Branch: master

commit df320474c5d07ecba0f0ac5cd73a6d5be66dc0a2
Author: Kevin Benton <email address hidden>
Date: Wed Apr 19 22:30:14 2017 -0700

    Set MTU on tap devices in Linux Bridge agent

    Libvirt does not set the MTU of the tap device it creates when creating
    a bridge interface. It also does not set the MTU of the bridge itself.
    This cannot be fixed on the Nova side since libvirt doesn't appear to
    have support for setting MTUs on bridges until version 3x.

    This results in a situation where the first VM tap interface attached to
    a bridge will always have an MTU of 1500. The Neutron agent will then
    add in VLAN/VXLAN interfaces with the correct MTU; however, the bridge
    MTU will still be pinned to the smallest interface MTU attached to it.
    This breaks jumbo frames until all small MTU tap devices are removed
    from the bridge.

    This patch explicitly sets the MTU on tap devices to match the network
    MTU when processing the device.

    We will have to carry this workaround until we stop Nova from
    plugging taps into bridges[1] or until we drop support for older
    libvirts on the Nova side and have it set the MTU.

    This bug was introduced by change
    I53c0eb57da956b36f09731d25db989719e9bc9dc which reverted automatic
    setting of tap MTUs to match those of the physical device.

    1. I23c5faaeab69aede1fd038a36f4a0b8f928498ce
    Closes-Bug: #1684038
    Change-Id: Ia245a3e22339fce026901e24a82e836c8b27cc28

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/459729

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459729
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c72b0595359d60f0d6b9f443908976c0b2201d2a
Submitter: Jenkins
Branch: stable/ocata

commit c72b0595359d60f0d6b9f443908976c0b2201d2a
Author: Kevin Benton <email address hidden>
Date: Wed Apr 19 22:30:14 2017 -0700

    Set MTU on tap devices in Linux Bridge agent

    Libvirt does not set the MTU of the tap device it creates when creating
    a bridge interface. It also does not set the MTU of the bridge itself.
    This cannot be fixed on the Nova side since libvirt doesn't appear to
    have support for setting MTUs on bridges until version 3x.

    This results in a situation where the first VM tap interface attached to
    a bridge will always have an MTU of 1500. The Neutron agent will then
    add in VLAN/VXLAN interfaces with the correct MTU; however, the bridge
    MTU will still be pinned to the smallest interface MTU attached to it.
    This breaks jumbo frames until all small MTU tap devices are removed
    from the bridge.

    This patch explicitly sets the MTU on tap devices to match the network
    MTU when processing the device.

    We will have to carry this workaround until we stop Nova from
    plugging taps into bridges[1] or until we drop support for older
    libvirts on the Nova side and have it set the MTU.

    This bug was introduced by change
    I53c0eb57da956b36f09731d25db989719e9bc9dc which reverted automatic
    setting of tap MTUs to match those of the physical device.

    1. I23c5faaeab69aede1fd038a36f4a0b8f928498ce
    Closes-Bug: #1684038
    Change-Id: Ia245a3e22339fce026901e24a82e836c8b27cc28
    (cherry picked from commit df320474c5d07ecba0f0ac5cd73a6d5be66dc0a2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/458293
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=233a68d27c0225bd461795c4418a9fed2fe11346
Submitter: Jenkins
Branch: master

commit 233a68d27c0225bd461795c4418a9fed2fe11346
Author: Kevin Benton <email address hidden>
Date: Wed Apr 19 16:16:41 2017 -0700

    Prevent regression of IP loss on MAC update

    This adjusts the UTs to ensure MAC address updates
    don't wipe out the original fixed IPs on a port if the
    IPs weren't part of the update.

    Change-Id: Ibd9ec8ece6678c837027b59c1154db872270e7be
    Related-Bug: #1684038

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/493652

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/493653

[Ihar Hrachyshka (ihar-hrachyshka)]

Ocata backport executed; as for Newton and earlier, those are not affected.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493652
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5cb46a79d5032c10950540039206c1cee4b801e8
Submitter: Jenkins
Branch: stable/ocata

commit 5cb46a79d5032c10950540039206c1cee4b801e8
Author: Kevin Benton <email address hidden>
Date: Wed Apr 19 16:16:41 2017 -0700

    Prevent regression of IP loss on MAC update

    This adjusts the UTs to ensure MAC address updates
    don't wipe out the original fixed IPs on a port if the
    IPs weren't part of the update.

    Change-Id: Ibd9ec8ece6678c837027b59c1154db872270e7be
    Related-Bug: #1684038
    (cherry picked from commit 233a68d27c0225bd461795c4418a9fed2fe11346)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493653
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b5594aa5fe4ea78b079233385a09fe7079af65fc
Submitter: Jenkins
Branch: stable/newton

commit b5594aa5fe4ea78b079233385a09fe7079af65fc
Author: Kevin Benton <email address hidden>
Date: Wed Apr 19 16:16:41 2017 -0700

    Prevent regression of IP loss on MAC update

    This adjusts the UTs to ensure MAC address updates
    don't wipe out the original fixed IPs on a port if the
    IPs weren't part of the update.

    Change-Id: Ibd9ec8ece6678c837027b59c1154db872270e7be
    Related-Bug: #1684038
    (cherry picked from commit 233a68d27c0225bd461795c4418a9fed2fe11346)