dhcp agent reporting state as down during the initial sync

[Daniel Alvarez]

When dhcp agent is started, neutron agent-list reports its state as dead until the initial sync is complete.

This can lead to unwanted alarms in monitoring systems, especially in large environments where the initial sync may take hours. During this time, systemctl shows that the agent is actually alive while neutron agent-list reports it as down.

Technical details:

If I'm right, this line [0] is the exact point where the initial sync takes place right after the first state report (with start_flag=True) is sent to the server. As it's being done in the same thread, it won't send a second state report until it's done with the sync.

Doing it in a separate thread would let the heartbeat task to continue sending state reports to the server but I don't know whether this have any unwanted side effects.

[0] https://github.com/openstack/neutron/blob/master/neutron/agent/dhcp/agent.py#L751

[Miguel Lavalle (minsel)]

@Daniel,

This seems like it may be a nice enhancement. Do you have specific information from large deployers that you could share here, so we can help the drivers to prioritize this enhancement?

[Daniel Alvarez (dalvarezs)]

@Miguel,
1200 tenants and about 4400 VMs take ~2.5 hours for dhcp-agent to sync

[Armando Migliaccio (armando-migliaccio)]

This seems a bug to me, but we would need to triage this one. Can you share version details, please?

[Daniel Alvarez (dalvarezs)]

@Armando, I agree with you. Also, it's present in master.
I was reviewing this patch [0] and I think it'll fix it (see my comments in gerrit)
Thanks, Daniel

[0] https://review.openstack.org/#/c/413010/

[Miguel Angel Ajo (mangelajo)]

Agreed, it's not an enhancement, but a bug fix. I've analyzed the change, talked to dalvarez, and +2'd it. I believe we should also backport this.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/413010
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f15851b98974dc16606da195cf3ecee577cd0ef8
Submitter: Jenkins
Branch: master

commit f15851b98974dc16606da195cf3ecee577cd0ef8
Author: Bertrand Lallau <email address hidden>
Date: Tue Dec 20 10:53:41 2016 +0100

    DHCP: enhance DHCPAgent startup procedure

    During DhcpAgent startup procedure all the following networks
    initialization is actually perform twice:
     * Killing old dnsmasq processes
     * set and configure all TAP interfaces
     * building all Dnsmasq config files (lease and host files)
     * launching dnsmasq processes
    What is done during the second iteration is just clean and redo
    exactly the same another time! This is really inefficient and
    increase dramatically DHCP startup time (near twice than needed).

    Initialization process 'sync_state' method is called twice:
     * one time during init_host()
     * another time during _report_state()

    sync_state() call must stay in init_host() due to bug #1420042.

    sync_state() is always called during startup in init_host()
    and will be periodically called by periodic_resync()
    to do reconciliation.
    Hence it can safely be removed from the run() method.

    Change-Id: Id6433598d5c833d2e86be605089d42feee57c257
    Closes-bug: #1651368
    Closes-Bug: #1650611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/422519

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/422519
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ae0a31d8b0c449031432117ac33c1e5bdf9d5957
Submitter: Jenkins
Branch: stable/newton

commit ae0a31d8b0c449031432117ac33c1e5bdf9d5957
Author: Bertrand Lallau <email address hidden>
Date: Tue Dec 20 10:53:41 2016 +0100

    DHCP: enhance DHCPAgent startup procedure

    During DhcpAgent startup procedure all the following networks
    initialization is actually perform twice:
     * Killing old dnsmasq processes
     * set and configure all TAP interfaces
     * building all Dnsmasq config files (lease and host files)
     * launching dnsmasq processes
    What is done during the second iteration is just clean and redo
    exactly the same another time! This is really inefficient and
    increase dramatically DHCP startup time (near twice than needed).

    Initialization process 'sync_state' method is called twice:
     * one time during init_host()
     * another time during _report_state()

    sync_state() call must stay in init_host() due to bug #1420042.

    sync_state() is always called during startup in init_host()
    and will be periodically called by periodic_resync()
    to do reconciliation.
    Hence it can safely be removed from the run() method.

    Change-Id: Id6433598d5c833d2e86be605089d42feee57c257
    Closes-bug: #1651368
    Closes-Bug: #1650611
    (cherry picked from commit f15851b98974dc16606da195cf3ecee577cd0ef8)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423206

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/424049

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/423206
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=21963618ca7accfe0b5d3389ee15426d5032a2ac
Submitter: Jenkins
Branch: stable/mitaka

commit 21963618ca7accfe0b5d3389ee15426d5032a2ac
Author: Bertrand Lallau <email address hidden>
Date: Tue Dec 20 10:53:41 2016 +0100

    DHCP: enhance DHCPAgent startup procedure

    During DhcpAgent startup procedure all the following networks
    initialization is actually perform twice:
     * Killing old dnsmasq processes
     * set and configure all TAP interfaces
     * building all Dnsmasq config files (lease and host files)
     * launching dnsmasq processes
    What is done during the second iteration is just clean and redo
    exactly the same another time! This is really inefficient and
    increase dramatically DHCP startup time (near twice than needed).

    Initialization process 'sync_state' method is called twice:
     * one time during init_host()
     * another time during _report_state()

    sync_state() call must stay in init_host() due to bug #1420042.

    sync_state() is always called during startup in init_host()
    and will be periodically called by periodic_resync()
    to do reconciliation.
    Hence it can safely be removed from the run() method.

    Change-Id: Id6433598d5c833d2e86be605089d42feee57c257
    Closes-bug: #1651368
    Closes-Bug: #1650611
    (cherry picked from commit f15851b98974dc16606da195cf3ecee577cd0ef8)
    (cherry picked from commit ae0a31d8b0c449031432117ac33c1e5bdf9d5957)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ihar Hrachyshka (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/424049