"network_namespace_exists"can timeout in loaded systems

[Rodolfo Alonso]

The method "neutron.agent.linux.ip_lib.network_namespace_exists" can timeout in loaded systems. We have detected this problem during some capacity tests. During those tests, the controllers are creating and deleting networks; that implies the creation and deletion of DHCP namespaces "qdhcp-xxx" in the same hosts. Adding additional logs (tagged with "SK;"), we found this method was timing out.

Snippet: https://paste.opendev.org/show/810132/

New additional logs added: https://paste.opendev.org/show/810133/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/814868

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/814868
Committed: https://opendev.org/openstack/neutron/commit/8127221479834350465251839ab88056f4f64fb4
Submitter: "Zuul (22348)"
Branch: master

commit 8127221479834350465251839ab88056f4f64fb4
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/816512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/816513

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/816515

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/816516

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/816518

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816515
Committed: https://opendev.org/openstack/neutron/commit/6a9c05a92498630b85102eb6af1eccb415fd605a
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 6a9c05a92498630b85102eb6af1eccb415fd605a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Conflicts:
          neutron/privileged/agent/linux/utils.py

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17
    (cherry picked from commit 8127221479834350465251839ab88056f4f64fb4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816513
Committed: https://opendev.org/openstack/neutron/commit/f95f2ed42a23e4ce300f0b8de726b4687c755020
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f95f2ed42a23e4ce300f0b8de726b4687c755020
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17
    (cherry picked from commit 8127221479834350465251839ab88056f4f64fb4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816512
Committed: https://opendev.org/openstack/neutron/commit/c05d07bd8f675bd17ee40ab10f5ccf90bdbdf8d2
Submitter: "Zuul (22348)"
Branch: stable/xena

commit c05d07bd8f675bd17ee40ab10f5ccf90bdbdf8d2
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17
    (cherry picked from commit 8127221479834350465251839ab88056f4f64fb4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816516
Committed: https://opendev.org/openstack/neutron/commit/f144ba95a59b156f41e4fd6333349b381251f79f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit f144ba95a59b156f41e4fd6333349b381251f79f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Conflicts:
          neutron/privileged/agent/linux/utils.py

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17
    (cherry picked from commit 8127221479834350465251839ab88056f4f64fb4)
    (cherry picked from commit 6a9c05a92498630b85102eb6af1eccb415fd605a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.4.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816518
Committed: https://opendev.org/openstack/neutron/commit/9cdd3310363b445e5751fc848e1354f7f52a0547
Submitter: "Zuul (22348)"
Branch: stable/train

commit 9cdd3310363b445e5751fc848e1354f7f52a0547
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Oct 21 08:49:41 2021 +0000

    Check a namespace existence by checking only its own directory

    To check the existance of a namespace, instead of listing the
    namespaces directory (by default "/var/run/netns"), this patch
    directly checks the existence of the namespace directory, using
    "os.path.exists".

    This check is faster than listing the whole directory and avoids
    timeout problems as reported in the related bug.

    Conflicts:
          neutron/privileged/agent/linux/utils.py

    Closes-Bug: #1947974
    Change-Id: I558d50d28378beb3710d98a2113ff9549c82ae17
    (cherry picked from commit 8127221479834350465251839ab88056f4f64fb4)
    (cherry picked from commit 6a9c05a92498630b85102eb6af1eccb415fd605a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 17.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.