Sometimes dnsmasq may not be restarted after adding new subnet

Bug #1848738 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Slawek Kaplonski

Bug Description

Tested on stable/queens but this code isn't changed a lot since than so probably may happen also on newer releases.

Sometimes (probably due to some race condition) dhcp agent spawns dnsmasq process and sets first "tag0" for the dhcp-range option and in opts file for subnet but later, when second subnet is added it may happend that this new subnet will "take" tag0 and old subnet should have "tag1" - it's based on subnets list order which is orderer based on subnet uuid.

Usually in such case dnsmasq process should be just restarted, tag in opts file should be changed and all should works fine.
But in some cases (quite rare) it may happen that tag is adjusted in opts file but not in dhcp-range command line option. Than such options are not send properly to the instances.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/689515

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/690582

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/689515
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bdc1bc73f3f118901295b247c9c7ce96511907ac
Submitter: Zuul
Branch: master

commit bdc1bc73f3f118901295b247c9c7ce96511907ac
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 18 07:04:43 2019 +0200

    Fix mismatch of tags in dnsmasq options

    In some cases (I don't know exactly how) it may happend that
    when new subnet, e.g. IPv6 is added to the network, subnets
    can change their order based on uuid.
    As before this patch we were using in dnsmasq options tags like
    "tagN" for subnets (where N was just number based on position of
    the subnet in the sorted list) it could happend sometimes that
    dnsmasq ended up with mismatch of tags configured in "dhcp-range"
    cmd option and set in "opts" file. That caused problem with serving
    proper DHCP options to the vms.

    This patch fixes this issue by using tags with format:
    "subnet-<uuid>" where uuid is id of the subnet. That was it's not
    based on order of subnets in the list and will always match with tag
    configured in opts file for specific subnet.

    As we was currently using port id as tag for "per port" DHCP options,
    this patch changes that to use tags like "port-<uuid>" to make it
    consistent with options configured "per subnet" and to make it easier
    to debug from where each option comes.

    Change-Id: Idaea33d62fa31edd7149ec916ec314438375724a
    Partial-Bug: #1848738

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/690719

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/690729

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/690731

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/690582
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88f2073526c2add7e8aa07656caec4c35c8b8459
Submitter: Zuul
Branch: stable/train

commit 88f2073526c2add7e8aa07656caec4c35c8b8459
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 18 07:04:43 2019 +0200

    Fix mismatch of tags in dnsmasq options

    In some cases (I don't know exactly how) it may happend that
    when new subnet, e.g. IPv6 is added to the network, subnets
    can change their order based on uuid.
    As before this patch we were using in dnsmasq options tags like
    "tagN" for subnets (where N was just number based on position of
    the subnet in the sorted list) it could happend sometimes that
    dnsmasq ended up with mismatch of tags configured in "dhcp-range"
    cmd option and set in "opts" file. That caused problem with serving
    proper DHCP options to the vms.

    This patch fixes this issue by using tags with format:
    "subnet-<uuid>" where uuid is id of the subnet. That was it's not
    based on order of subnets in the list and will always match with tag
    configured in opts file for specific subnet.

    As we was currently using port id as tag for "per port" DHCP options,
    this patch changes that to use tags like "port-<uuid>" to make it
    consistent with options configured "per subnet" and to make it easier
    to debug from where each option comes.

    Change-Id: Idaea33d62fa31edd7149ec916ec314438375724a
    Partial-Bug: #1848738

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/690719
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a0730e684d54796fce0574bb314b679bcdd58c76
Submitter: Zuul
Branch: stable/stein

commit a0730e684d54796fce0574bb314b679bcdd58c76
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 18 07:04:43 2019 +0200

    Fix mismatch of tags in dnsmasq options

    In some cases (I don't know exactly how) it may happend that
    when new subnet, e.g. IPv6 is added to the network, subnets
    can change their order based on uuid.
    As before this patch we were using in dnsmasq options tags like
    "tagN" for subnets (where N was just number based on position of
    the subnet in the sorted list) it could happend sometimes that
    dnsmasq ended up with mismatch of tags configured in "dhcp-range"
    cmd option and set in "opts" file. That caused problem with serving
    proper DHCP options to the vms.

    This patch fixes this issue by using tags with format:
    "subnet-<uuid>" where uuid is id of the subnet. That was it's not
    based on order of subnets in the list and will always match with tag
    configured in opts file for specific subnet.

    As we was currently using port id as tag for "per port" DHCP options,
    this patch changes that to use tags like "port-<uuid>" to make it
    consistent with options configured "per subnet" and to make it easier
    to debug from where each option comes.

    Conflicts:
        neutron/tests/unit/agent/linux/test_dhcp.py

    Change-Id: Idaea33d62fa31edd7149ec916ec314438375724a
    Partial-Bug: #1848738
    (cherry picked from commit 88f2073526c2add7e8aa07656caec4c35c8b8459)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/690729
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ba12b9e36916de4f4635de530defcbffc6dc1a53
Submitter: Zuul
Branch: stable/rocky

commit ba12b9e36916de4f4635de530defcbffc6dc1a53
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 18 07:04:43 2019 +0200

    Fix mismatch of tags in dnsmasq options

    In some cases (I don't know exactly how) it may happend that
    when new subnet, e.g. IPv6 is added to the network, subnets
    can change their order based on uuid.
    As before this patch we were using in dnsmasq options tags like
    "tagN" for subnets (where N was just number based on position of
    the subnet in the sorted list) it could happend sometimes that
    dnsmasq ended up with mismatch of tags configured in "dhcp-range"
    cmd option and set in "opts" file. That caused problem with serving
    proper DHCP options to the vms.

    This patch fixes this issue by using tags with format:
    "subnet-<uuid>" where uuid is id of the subnet. That was it's not
    based on order of subnets in the list and will always match with tag
    configured in opts file for specific subnet.

    As we was currently using port id as tag for "per port" DHCP options,
    this patch changes that to use tags like "port-<uuid>" to make it
    consistent with options configured "per subnet" and to make it easier
    to debug from where each option comes.

    Conflicts:
        neutron/agent/linux/dhcp.py
        neutron/tests/unit/agent/linux/test_dhcp.py

    Change-Id: Idaea33d62fa31edd7149ec916ec314438375724a
    Partial-Bug: #1848738
    (cherry picked from commit 88f2073526c2add7e8aa07656caec4c35c8b8459)
    (cherry picked from commit a0730e684d54796fce0574bb314b679bcdd58c76)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/690731
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1f7cd398d675400494b77371f107a0bd2c3b2e20
Submitter: Zuul
Branch: stable/queens

commit 1f7cd398d675400494b77371f107a0bd2c3b2e20
Author: Slawek Kaplonski <email address hidden>
Date: Fri Oct 18 07:04:43 2019 +0200

    Fix mismatch of tags in dnsmasq options

    In some cases (I don't know exactly how) it may happend that
    when new subnet, e.g. IPv6 is added to the network, subnets
    can change their order based on uuid.
    As before this patch we were using in dnsmasq options tags like
    "tagN" for subnets (where N was just number based on position of
    the subnet in the sorted list) it could happend sometimes that
    dnsmasq ended up with mismatch of tags configured in "dhcp-range"
    cmd option and set in "opts" file. That caused problem with serving
    proper DHCP options to the vms.

    This patch fixes this issue by using tags with format:
    "subnet-<uuid>" where uuid is id of the subnet. That was it's not
    based on order of subnets in the list and will always match with tag
    configured in opts file for specific subnet.

    As we was currently using port id as tag for "per port" DHCP options,
    this patch changes that to use tags like "port-<uuid>" to make it
    consistent with options configured "per subnet" and to make it easier
    to debug from where each option comes.

    Conflicts:
        neutron/agent/linux/dhcp.py
        neutron/tests/unit/agent/linux/test_dhcp.py

    Change-Id: Idaea33d62fa31edd7149ec916ec314438375724a
    Partial-Bug: #1848738
    (cherry picked from commit 88f2073526c2add7e8aa07656caec4c35c8b8459)
    (cherry picked from commit a0730e684d54796fce0574bb314b679bcdd58c76)
    (cherry picked from commit ba12b9e36916de4f4635de530defcbffc6dc1a53)

tags: added: in-stable-queens
Changed in neutron:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.