Slow fact gathering until network configuration is applied

[Pierre Riteau]

In an environment without Internet access, we are seeing `kayobe overcloud host configure` wait for many seconds during each Gathering Facts task, until the network configuration step is reached. Once network configuration is applied by Kayobe, TASK [Gathering Facts] takes only a fraction of a second.

This is because the configuration applied by the config drive configures the seed host as default gateway, but it cannot be used to reach the Internet. I assume the slowness is caused by attempts to resolve hostnames (DNS servers are left to the default Google DNS).

Seen on Yoga release with Rocky Linux 9.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kayobe/+/898434

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/898434
Committed: https://opendev.org/openstack/kayobe/commit/9980a8f891edac8f21d0cde9ee1e756a40c7be3b
Submitter: "Zuul (22348)"
Branch: master

commit 9980a8f891edac8f21d0cde9ee1e756a40c7be3b
Author: Pierre Riteau <email address hidden>
Date: Mon Oct 16 17:31:05 2023 +0200

    Resolve slow fact gathering when seed SNAT is off

    Stop configuring the seed host as gateway when seed SNAT is disabled
    (which is the default since Xena), otherwise overcloud hosts may take a
    long time to gather facts until their network configuration is applied
    (probably because DNS servers are thought to be reachable until the
    gateway is changed or removed).

    Closes-Bug: #2039461
    Change-Id: Ib847d3420dee374cec067cd8af519b510be04120

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kayobe/+/899048

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kayobe/+/899049

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kayobe/+/899050

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/899050
Committed: https://opendev.org/openstack/kayobe/commit/8a1aa4dc9bcbb22a8189b2bda315c0d55074b76b
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 8a1aa4dc9bcbb22a8189b2bda315c0d55074b76b
Author: Pierre Riteau <email address hidden>
Date: Mon Oct 16 17:31:05 2023 +0200

    Resolve slow fact gathering when seed SNAT is off

    Stop configuring the seed host as gateway when seed SNAT is disabled
    (which is the default since Xena), otherwise overcloud hosts may take a
    long time to gather facts until their network configuration is applied
    (probably because DNS servers are thought to be reachable until the
    gateway is changed or removed).

    Closes-Bug: #2039461
    Change-Id: Ib847d3420dee374cec067cd8af519b510be04120
    (cherry picked from commit 9980a8f891edac8f21d0cde9ee1e756a40c7be3b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/899048
Committed: https://opendev.org/openstack/kayobe/commit/af59e6d953326904eed034ef6cc19dd54b82b805
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit af59e6d953326904eed034ef6cc19dd54b82b805
Author: Pierre Riteau <email address hidden>
Date: Mon Oct 16 17:31:05 2023 +0200

    Resolve slow fact gathering when seed SNAT is off

    Stop configuring the seed host as gateway when seed SNAT is disabled
    (which is the default since Xena), otherwise overcloud hosts may take a
    long time to gather facts until their network configuration is applied
    (probably because DNS servers are thought to be reachable until the
    gateway is changed or removed).

    Closes-Bug: #2039461
    Change-Id: Ib847d3420dee374cec067cd8af519b510be04120
    (cherry picked from commit 9980a8f891edac8f21d0cde9ee1e756a40c7be3b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kayobe/+/899049
Committed: https://opendev.org/openstack/kayobe/commit/e5876e386844219ae47549e39a02e367e9faf248
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e5876e386844219ae47549e39a02e367e9faf248
Author: Pierre Riteau <email address hidden>
Date: Mon Oct 16 17:31:05 2023 +0200

    Resolve slow fact gathering when seed SNAT is off

    Stop configuring the seed host as gateway when seed SNAT is disabled
    (which is the default since Xena), otherwise overcloud hosts may take a
    long time to gather facts until their network configuration is applied
    (probably because DNS servers are thought to be reachable until the
    gateway is changed or removed).

    Closes-Bug: #2039461
    Change-Id: Ib847d3420dee374cec067cd8af519b510be04120
    (cherry picked from commit 9980a8f891edac8f21d0cde9ee1e756a40c7be3b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kayobe 15.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kayobe 14.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kayobe 12.10.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kayobe 13.4.0 release.