Train -> Ussuri undercloud upgrade failing with on pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '192.168.24.3' ([Errno 113] No route to host)")

[Sandeep Yadav]

Description:-

Train -> Ussuri undercloud upgrade failing with on pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '192.168.24.3' ([Errno 113] No route to host)")

Log snippet:-

https://4a5f7324f50c525e768c-c393df920c3ba3d74686e2bffd0842f4.ssl.cf2.rackcdn.com/792084/1/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/ab32dd4/logs/undercloud/home/zuul/undercloud_upgrade.log
~~~
2021-05-30 01:20:41 | 2021-05-30 01:20:41.541278 | fa163e91-7420-991c-9f4c-00000000329b | TASK | Check containers status
2021-05-30 01:26:03 | [ERROR]: Container(s) which finished with wrong return code:
2021-05-30 01:26:03 | ['mistral_db_sync', 'glance_api_db_sync']
2021-05-30 01:26:03 | [ERROR]: Container(s) which failed to be created by podman_container module:
2021-05-30 01:26:03 | ['keystone_db_sync', 'heat_engine_db_sync']
2021-05-30 01:26:03 | [ERROR]: Container(s) which did not finish after 300 minutes:
2021-05-30 01:26:03 | ['keystone_db_sync', 'heat_engine_db_sync']
2021-05-30 01:26:03 | 2021-05-30 01:26:03.365437 | fa163e91-7420-991c-9f4c-00000000329b | FATAL | Check containers status | undercloud | error={"changed": false, "msg": "Failed container(s): ['keystone_db_sync', 'heat_engine_db_sync', 'mistral_db_sync', 'glance_api_db_sync'], check logs in /var/log/containers/stdouts/"}
~~~

https://4a5f7324f50c525e768c-c393df920c3ba3d74686e2bffd0842f4.ssl.cf2.rackcdn.com/792084/1/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/ab32dd4/logs/undercloud/var/log/extra/podman/containers/mistral_db_sync/stdout.log

~~~
CRITI [Mistral] Unhandled error
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pymysql/connections.py", line 571, in connect
    **kwargs)
  File "/usr/lib64/python3.6/socket.py", line 724, in create_connection
    raise err
  File "/usr/lib64/python3.6/socket.py", line 713, in create_connection
    sock.connect(sa)
OSError: [Errno 113] No route to host

During handling of the above exception, another exception occurred:
.
.
sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.3' ([Errno 113] No route to host)")
(Background on this error at: http://sqlalche.me/e/e3q8)
~~~

https://4a5f7324f50c525e768c-c393df920c3ba3d74686e2bffd0842f4.ssl.cf2.rackcdn.com/792084/1/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/ab32dd4/logs/undercloud/var/log/extra/podman/containers/heat_engine_db_sync/stdout.log

~~~
2021-05-30 01:38:58.414 7 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -190 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.3' ([Errno 113] No route to host)")
~~~

Another example:

https://7f0dbc48bcc1a58e17d2-35e520be6a56e06b3e25b38121a22180.ssl.cf1.rackcdn.com/793277/1/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/c22ba14/logs/undercloud/home/zuul/undercloud_upgrade.log

https://7f0dbc48bcc1a58e17d2-35e520be6a56e06b3e25b38121a22180.ssl.cf1.rackcdn.com/793277/1/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/c22ba14/logs/undercloud/var/log/extra/podman/containers/mistral_db_sync/stdout.log

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/793693

[Sandeep Yadav (sandeepyadav93)]

https://review.opendev.org/c/openstack/tripleo-ci/+/793693

~~~
Set T->U undercloud upgrade to non-voting

tripleo-ci-centos-8-undercloud-upgrade-ussuri is failing with issue
as mentioned in Related-Bug.

Setting this job to non-voting and removing from gate jobs to
unblock check/gate.
~~~

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/793693
Committed: https://opendev.org/openstack/tripleo-ci/commit/3f9eb85616ac96d29135d930c380afd420527457
Submitter: "Zuul (22348)"
Branch: master

commit 3f9eb85616ac96d29135d930c380afd420527457
Author: Sandeep Yadav <email address hidden>
Date: Sun May 30 14:43:41 2021 +0530

    Set T->U undercloud upgrade to non-voting

    tripleo-ci-centos-8-undercloud-upgrade-ussuri is failing with issue
    as mentioned in Related-Bug.

    Setting this job to non-voting and removing from gate jobs to
    unblock check/gate.

    Related-Bug: #1930151
    Change-Id: I5d694c551a5aa5a34eda135520eb2e1bfebd73ed

[Marios Andreou (marios-b)]

this has been happening since ~ 21st May. That is when the last green runs were in [1] and I found an example of the bug from the 21st there [2]

        * 2021-05-21 15:58:52 | 2021-05-21 15:58:52.602476 | fa163e9b-7524-3e5f-0671-000000003278 | FATAL | Check containers status | undercloud | error={"changed": false, "msg": "Failed container(s): ['nova_api_db_sync', 'zaqar_db_sync', 'keystone_db_sync', 'neutron_db_sync'], check logs in /var/log/containers/stdouts/"}

[1] https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade-ussuri
[2] https://0bdfd6862a2eb7fa74ee-67e15cc69bd7500d8d48cf199d18d0c8.ssl.cf5.rackcdn.com/782772/2/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/809e547/logs/undercloud/home/zuul/undercloud_upgrade.log

[Marios Andreou (marios-b)]

picked this up for digging today - no root cause yet but some notes for now:

-------------------
1. Don't think this is anything to do with mistral.

The trace in the description mentions mistral but we have a similar error there [1] for nova_api_db_sync.log: https://0bdfd6862a2eb7fa74ee-67e15cc69bd7500d8d48cf199d18d0c8.ssl.cf5.rackcdn.com/782772/2/gate/tripleo-ci-centos-8-undercloud-upgrade-ussuri/809e547/logs/undercloud/var/log/containers/stdouts/nova_api_db_sync.log

Also, looking at the mistral migration in question (where it fails):

 INFO [alembic.runtime.migration] Running upgrade 033 -> 034, Create scheduled jobs table.
 ->EXPLOSION

It doesn't appear to have been touched in a long time: https://opendev.org/openstack/mistral/commits/branch/stable/ussuri/mistral/db/sqlalchemy/migration/alembic_migrations/versions/034_add_scheduled_jobs_table.py

-------------------
2. Comparing package versions didn't spot something obvious:

 * (green/good job) https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_28b/792331/2/check/tripleo-ci-centos-8-undercloud-upgrade-ussuri/28bb7df/logs/undercloud/var/log/dnf.log
  2021-05-21T09:53:51+0000 DEBUG ---> Package rdo-openvswitch.noarch 1:2.13-2.el8 will be an upgrade
  2021-05-21T09:02:49+0000 DEBUG Installed: python3-PyMySQL-0.10.1-2.module_el8.5.0+761+faacb0fb.noarch
  2021-05-21T09:02:49+0000 DEBUG Installed: puppet-mysql-10.4.1-0.20200530020255.95f9b98.el8.noarch
  2021-05-21T09:55:27+0000 DEBUG Upgraded: puppet-ovn-16.3.1-0.20210331142948.74d2389.el8.noarch

        * (failed job) https://8f87caa881e8eda0ef3b-848f29e6bb7f92a80e59238721ad95f7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-undercloud-upgrade-ussuri/75847b1/logs/undercloud/var/log/dnf.log
  2021-06-16T08:13:25+0000 DEBUG ---> Package rdo-openvswitch.noarch 1:2.13-3.el8 will be an upgrade
  2021-06-16T07:21:59+0000 DEBUG ---> Package python3-PyMySQL.noarch 0.10.1-2.module_el8.5.0+761+faacb0fb will be installed
  2021-06-16T07:21:59+0000 DEBUG ---> Package puppet-mysql.noarch 10.4.1-0.20200530020255.95f9b98.el8 will be installed
  2021-06-16T08:14:49+0000 DEBUG Upgraded: puppet-ovn-16.3.1-0.20210331142948.74d2389.el8.noarch

-------------------
3. Something happened/merged ~ 21st May that caused this

Last successful runs then https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-undercloud-upgrade-ussuri&result=SUCCESS
I have tried checking tripleo-heat-templates, tripleo-common, python-tripleoclient and tripleo-upgrade but didn't spot something yet (e.g. git log --no-merges --since "May 21 2021" --pretty=format:'%C(auto)%h%d (%ci) %cn <%ce> %s' )

[Cédric Jeanneret (cjeanner)]

Wondering if this isn't linked to the move from xinetd -> socat within clustercheck. Would be interesting to ensure that service is up'n'running... ?

This might be of interest for PIDONE.

[Cédric Jeanneret (cjeanner)]

so, nope, not related:
that move is for W, and we're on the UC, no pacemaker there anyway.

haproxy shows mysql backend as up 2 minutes before the crash[0] - and, fun thing, it seems the mistral_db_sync is able to access the DB for a while and then, crash:

Jun 16 08:29:52 undercloud haproxy[7]: Backup Server mysql/undercloud.ctlplane.localdomain is UP, reason: Layer4 check passed, check duration: 0ms. 0 active and 1 backup servers online. Running on backup. 0 sessions requeued, 0 total in queue.

When the mistral_db_sync crashes, we can see the following time: 08:32:01

There is no mysql activity around that time in haproxy log[1].

[0] https://8f87caa881e8eda0ef3b-848f29e6bb7f92a80e59238721ad95f7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-undercloud-upgrade-ussuri/75847b1/logs/undercloud/var/log/containers/stdouts/mistral_db_sync.log
[1] https://8f87caa881e8eda0ef3b-848f29e6bb7f92a80e59238721ad95f7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-undercloud-upgrade-ussuri/75847b1/logs/undercloud/var/log/containers/haproxy/haproxy.log

[Alex Schultz (alex-schultz)]

192.168.24.3 is missing from the host

https://8f87caa881e8eda0ef3b-848f29e6bb7f92a80e59238721ad95f7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/tripleo-heat-templates/stable/ussuri/tripleo-ci-centos-8-undercloud-upgrade-ussuri/75847b1/logs/undercloud/var/log/extra/network.txt
### IPv4 addresses
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 198.72.124.14/24 brd 198.72.124.255 scope global noprefixroute ens3
       valid_lft forever preferred_lft forever
4: br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1350 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.24.2/24 scope global br-ex
       valid_lft forever preferred_lft forever
5: br-ctlplane: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.24.1/24 brd 192.168.24.255 scop

[Alex Schultz (alex-schultz)]

https://review.opendev.org/c/openstack/tripleo-ansible/+/792032 is likely the cause. we're not updating the network config and rerunning os-net-config when we upgrade. We need to for T->U because the vips are defined in the NetworkConfig in U but not in T (we ditched keepalived)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796712

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796728

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796729

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796730

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796731

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796728
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/c2966ca0fb4a477492031275d792e3c5f8b6b5a5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit c2966ca0fb4a477492031275d792e3c5f8b6b5a5
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Related-Bug: #1930151
    (cherry picked from commit 38ca24be2feda9baac7f5ddff221dcb5b2222240)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796731
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/ec1a78829a9db51e1140c5bae68500bf6e94e3b5
Submitter: "Zuul (22348)"
Branch: stable/train

commit ec1a78829a9db51e1140c5bae68500bf6e94e3b5
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Related-Bug: #1930151
    (cherry picked from commit 38ca24be2feda9baac7f5ddff221dcb5b2222240)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796730
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/4efcf186259efd22ae56357bd378e32231a0f880
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 4efcf186259efd22ae56357bd378e32231a0f880
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Related-Bug: #1930151
    (cherry picked from commit 38ca24be2feda9baac7f5ddff221dcb5b2222240)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796729
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/c2c2111ef5aaef12e3a9fe70f4569e606de2cfe6
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c2c2111ef5aaef12e3a9fe70f4569e606de2cfe6
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Related-Bug: #1930151
    (cherry picked from commit 38ca24be2feda9baac7f5ddff221dcb5b2222240)

[Marios Andreou (marios-b)]

@Alex we will need something further here - see comment at https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796730/1#message-02f7d17e8befd7a111a65110bebcfe7c4b1bb27f on the ussuri patch the job was still failing in the same way (it is non voting because of this bug and so it didn't gate that fix).

[Marios Andreou (marios-b)]

Alex has reverted those:

wallaby * https://review.opendev.org/q/c2966ca0fb4a477492031275d792e3c5f8b6b5a5
train * https://review.opendev.org/q/ec1a78829a9db51e1140c5bae68500bf6e94e3b5

and since master didn't merge yet updated v2 @ https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796712/2/environments/undercloud.yaml#82 with the 'proper' fix (s/NetworkDeploymentAction/NetworkDeploymentActions)

once it merges it can be picked back to ussuri again.

[Marios Andreou (marios-b)]

BTW does anyone know how this happened - I noticed while digging for comment #20.

We have two identical commits (the revert) on wallaby:

        * https://opendev.org/openstack/tripleo-heat-templates/commit/a15698b7f91156dbaa9b35afe5b49d6df4e182b0

        * https://opendev.org/openstack/tripleo-heat-templates/commit/2d095f314d7c6552b967fbd9eff906acb4f5de1a

Did we break git? :D

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/796712
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/3b043e4953393c7723553c84cc094a25902cd083
Submitter: "Zuul (22348)"
Branch: master

commit 3b043e4953393c7723553c84cc094a25902cd083
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Related-Bug: #1930151

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797493

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797494

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797495

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797496

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ci/+/797769

[Marios Andreou (marios-b)]

@Alex the ussuri patch in particular needs some attention as that is where we will see if the problem is fixed.

With the latest version there is still something further needed https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797496/1#message-516ade87bb34a8ffbefe3ef2efce357147d69dd8

I posted the revert of 'make undercloud-upgrade ussuri non voting' there https://review.opendev.org/c/openstack/tripleo-ci/+/797769 - ideally we should make your ussuri patch depends-on this one so we are sure we have the fix before it merges.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797496
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/21c696d2d64d354f3f179e9d1960cc90cb85ed5f
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 21c696d2d64d354f3f179e9d1960cc90cb85ed5f
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    NOTE: Changed Change-Id as we incorrectly early cherry-picked and
    and to revert.
    Original-Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Change-Id: Id3924a027c9bd7865c9fb081a2eda4f8be069a57
    Related-Bug: #1930151
    (cherry picked from commit 3b043e4953393c7723553c84cc094a25902cd083)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-ci/+/797769
Committed: https://opendev.org/openstack/tripleo-ci/commit/232de9dd839755601648af65385553956991e634
Submitter: "Zuul (22348)"
Branch: master

commit 232de9dd839755601648af65385553956991e634
Author: Marios Andreou <email address hidden>
Date: Thu Jun 24 11:03:06 2021 +0000

    Revert "Set T->U undercloud upgrade to non-voting"

    This reverts commit 3f9eb85616ac96d29135d930c380afd420527457.

    Related-Bug: 1930151
    Reason for revert: Job should be fixed and made voting again

    Change-Id: I21450e262b674c795e556e04c18e2627cb3a410f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797495
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/6ba7307e6a94d2f339b945817cce620d42449fab
Submitter: "Zuul (22348)"
Branch: stable/train

commit 6ba7307e6a94d2f339b945817cce620d42449fab
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    NOTE: Changed Change-Id as we incorrectly early cherry-picked and
    and to revert.
    Original-Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Change-Id: Id3924a027c9bd7865c9fb081a2eda4f8be069a57
    Related-Bug: #1930151
    (cherry picked from commit 3b043e4953393c7723553c84cc094a25902cd083)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797494
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/c748d92c768ccf59d4579f35f8b14cda142686f2
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c748d92c768ccf59d4579f35f8b14cda142686f2
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    NOTE: Changed Change-Id as we incorrectly early cherry-picked and
    and to revert.
    Original-Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Change-Id: Id3924a027c9bd7865c9fb081a2eda4f8be069a57
    Related-Bug: #1930151
    (cherry picked from commit 3b043e4953393c7723553c84cc094a25902cd083)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/797493
Committed: https://opendev.org/openstack/tripleo-heat-templates/commit/d098242f778d823e944798afb083b85ea492cba3
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d098242f778d823e944798afb083b85ea492cba3
Author: Alex Schultz <email address hidden>
Date: Wed Jun 16 11:02:23 2021 -0600

    Always run network config for undercloud

    We need to ensure that os-net-config is run on upgrades. On the
    undercloud we want the os-net-config to always run so let's set the
    NetworkDeploymentAction correctly.

    NOTE: Changed Change-Id as we incorrectly early cherry-picked and
    and to revert.
    Original-Change-Id: I749714c1f0af3787020f3e9b88f752d272b91d37
    Change-Id: Id3924a027c9bd7865c9fb081a2eda4f8be069a57
    Related-Bug: #1930151
    (cherry picked from commit 3b043e4953393c7723553c84cc094a25902cd083)