[DVR][DB] too many slow query during agent restart

[LIU Yulong]

ENV: stable/queens

In a large scale cloud deployment, when restart neutron agent, especially l3 agents, neutron server side will trigger too many slow DB query. And this will cause the agent restart time to be too long to operate.

[Bence Romsics (bence-romsics)]

Hi Liu,

Do you have some numbers? How many agents, ports, routers you have when it becomes too slow? What time does the agent restart take? Is it just slow but eventually everything starts up properly? Or does it hit timeouts and do you see failures after the timeouts? Did you trace the DB queries? Which ones were slow?

If you can, please write about how to reproduce the problem you're seeing. Can it be reproduced in devstack or do you need lots of hardware for reproduction?

Cheers,
Bence

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/673557

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673557
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=dd96f3775939dc0209396b4bd73038c1962e6a7e
Submitter: Zuul
Branch: master

commit dd96f3775939dc0209396b4bd73038c1962e6a7e
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 18:06:22 2019 +0800

    Optimize DVR related port DB query

    Save order by in port query when not require fixed_ips,
    and save some useless query for dvr subnet mac.

    Closes-Bug: #1834308
    Change-Id: I6836840edcaa5a21fd2ba9f65ffd24f7e5038fa3

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/716304

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/716309

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/716310

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716304
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2d0adf4a0554e7f8b06a2aaa37d1e772b301de95
Submitter: Zuul
Branch: stable/stein

commit 2d0adf4a0554e7f8b06a2aaa37d1e772b301de95
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 18:06:22 2019 +0800

    Optimize DVR related port DB query

    Save order by in port query when not require fixed_ips,
    and save some useless query for dvr subnet mac.

    Closes-Bug: #1834308
    Change-Id: I6836840edcaa5a21fd2ba9f65ffd24f7e5038fa3
    (cherry picked from commit dd96f3775939dc0209396b4bd73038c1962e6a7e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716310
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=008051325eca146b79ec6c46cc6d37fc9c9ca2c8
Submitter: Zuul
Branch: stable/queens

commit 008051325eca146b79ec6c46cc6d37fc9c9ca2c8
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 18:06:22 2019 +0800

    Optimize DVR related port DB query

    Save order by in port query when not require fixed_ips,
    and save some useless query for dvr subnet mac.

    Conflicts:
        neutron/db/db_base_plugin_common.py
        neutron/db/l3_dvr_db.py

    Closes-Bug: #1834308
    Change-Id: I6836840edcaa5a21fd2ba9f65ffd24f7e5038fa3
    (cherry picked from commit dd96f3775939dc0209396b4bd73038c1962e6a7e)
    (cherry picked from commit 2d0adf4a0554e7f8b06a2aaa37d1e772b301de95)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716309
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=827546a0fb936cba846e29dbefecd78d49e9f745
Submitter: Zuul
Branch: stable/rocky

commit 827546a0fb936cba846e29dbefecd78d49e9f745
Author: LIU Yulong <email address hidden>
Date: Mon Jun 24 18:06:22 2019 +0800

    Optimize DVR related port DB query

    Save order by in port query when not require fixed_ips,
    and save some useless query for dvr subnet mac.

    Conflicts:
        neutron/db/db_base_plugin_common.py
        neutron/db/l3_dvr_db.py

    Closes-Bug: #1834308
    Change-Id: I6836840edcaa5a21fd2ba9f65ffd24f7e5038fa3
    (cherry picked from commit dd96f3775939dc0209396b4bd73038c1962e6a7e)
    (cherry picked from commit 2d0adf4a0554e7f8b06a2aaa37d1e772b301de95)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.