empty /etc/os-net-config/config.json is causing network disruptions

[Bob Fournier]

In Queens, os-apply-config is running os-net-config with an empty /etc/os-net-config/config.json file, overwriting the network configuration. Since the new style of nic configs is used in Queens which rely on the run-os-net-config.sh script, not on os-apply-config, its not necessary to have the file /usr/libexec/os-apply-config/templates/etc/os-net-config/.

Here is os-net-config being invoked by os-apply-config after the network has been configured:
messages:Mar 22 13:59:09 localhost os-collect-config: [2018/03/22 01:58:36 PM] [INFO] Using config file at: /etc/os-net-config/config.json
messages:Mar 22 13:59:20 localhost os-collect-config: [2018/03/22 01:59:20 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:30 localhost os-collect-config: [2018/03/22 01:59:30 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:43 localhost os-collect-config: [2018/03/22 01:59:43 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 13:59:55 localhost os-collect-config: [2018/03/22 01:59:55 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:18 localhost os-collect-config: [2018/03/22 02:00:18 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:33 localhost os-collect-config: [2018/03/22 02:00:33 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:00:46 localhost os-collect-config: [2018/03/22 02:00:46 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:01:03 localhost os-collect-config: [2018/03/22 02:01:03 PM] [INFO] writing /etc/os-net-config/config.json
messages:Mar 22 14:01:23 localhost os-collect-config: [2018/03/22 02:01:23 PM] [INFO] writing /etc/os-net-config/config.json

I workaround to remove this file has resolved the problem:
virt-customize -a overcloud-full.qcow2 --delete /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/555452

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/556539

[Marios Andreou (marios-b)]

this is also discussed at https://bugzilla.redhat.com/show_bug.cgi?id=1559151

also see another related fix at https://review.openstack.org/#/c/556533/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/555452
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=20e5f2766cf97c463fc73175ddcf31a1d7fb558d
Submitter: Zuul
Branch: master

commit 20e5f2766cf97c463fc73175ddcf31a1d7fb558d
Author: Bob Fournier <email address hidden>
Date: Thu Mar 22 16:48:29 2018 -0400

    Remove /etc/os-net-config/config.json in os-apply-config

    In the new style of nic config files, os-net-config is invoked by
    the script run-os-net-config.sh, not by os-apply-config. This is the
    only format supported in Queens. Remove the os-apply-config version
    of /etc/os-net-config/config.json to prevent os-apply-config from
    attempting to run with an empty config.json file.

    Note that /etc/os-net-config/config.json no longer exists in
    tripleo-image-elements.

    Change-Id: Ica86382c4e5a1463ed8b2146e3ec490b7e2c9452
    Closes-Bug: 1758161

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/556997

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/557739

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556533
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=55864c9e75b8ab29bb4bdae6ee4bf44ed7f9e419
Submitter: Zuul
Branch: master

commit 55864c9e75b8ab29bb4bdae6ee4bf44ed7f9e419
Author: mandreou <email address hidden>
Date: Mon Mar 26 16:22:48 2018 +0300

    Remove the os-net-config element config.json template

    On upgrade from environments earlier than Pike we should remove
    this file as it can cause problems with post-upgrade reboots
    see bug for more info.

    Change-Id: If7f4bef3a79a881f003ee42f62fe51c537004e2a
    Related-Bug: 1758161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/557739
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2113bf3ae9554b9c1cc3ab1cac9223b4022a56d2
Submitter: Zuul
Branch: stable/queens

commit 2113bf3ae9554b9c1cc3ab1cac9223b4022a56d2
Author: mandreou <email address hidden>
Date: Mon Mar 26 16:22:48 2018 +0300

    Remove the os-net-config element config.json template

    On upgrade from environments earlier than Pike we should remove
    this file as it can cause problems with post-upgrade reboots
    see bug for more info.

    Change-Id: If7f4bef3a79a881f003ee42f62fe51c537004e2a
    Related-Bug: 1758161
    (cherry picked from commit 55864c9e75b8ab29bb4bdae6ee4bf44ed7f9e419)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/557939

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556997
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=6f5d0b71552e48862e74ef242a2bbbd9f86bb1ec
Submitter: Zuul
Branch: stable/queens

commit 6f5d0b71552e48862e74ef242a2bbbd9f86bb1ec
Author: Bob Fournier <email address hidden>
Date: Thu Mar 22 16:48:29 2018 -0400

    Remove /etc/os-net-config/config.json in os-apply-config

    In the new style of nic config files, os-net-config is invoked by
    the script run-os-net-config.sh, not by os-apply-config. This is the
    only format supported in Queens. Remove the os-apply-config version
    of /etc/os-net-config/config.json to prevent os-apply-config from
    attempting to run with an empty config.json file.

    Note that /etc/os-net-config/config.json no longer exists in
    tripleo-image-elements.

    Change-Id: Ica86382c4e5a1463ed8b2146e3ec490b7e2c9452
    Closes-Bug: 1758161
    (cherry picked from commit 20e5f2766cf97c463fc73175ddcf31a1d7fb558d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556539
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9f4c691ccae4be6aa93e40450717b35f16a2a655
Submitter: Zuul
Branch: master

commit 9f4c691ccae4be6aa93e40450717b35f16a2a655
Author: Bob Fournier <email address hidden>
Date: Mon Mar 26 10:09:08 2018 -0400

    Remove config.json for old style nic config files when using new style

    This removes the config.json used by os-apply-config for old style nic
    config files in the script invoked for new style nic config files.
    This will avoid the case when the network files are set up by
    run-os-net-config.sh and then later overwritten by os-apply-config.

    Change-Id: I8f17a53c392aac657b00afdecbb5e58a2e192787
    Closes-Bug: 1758161

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/560022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/557939
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c5f6f704ac66f8100caa2eb55844ef4989879c7d
Submitter: Zuul
Branch: stable/pike

commit c5f6f704ac66f8100caa2eb55844ef4989879c7d
Author: mandreou <email address hidden>
Date: Mon Mar 26 16:22:48 2018 +0300

    Remove the os-net-config element config.json template

    On upgrade from environments earlier than Pike we should remove
    this file as it can cause problems with post-upgrade reboots
    see bug for more info.

    Change-Id: If7f4bef3a79a881f003ee42f62fe51c537004e2a
    Related-Bug: 1758161
    (cherry picked from commit 55864c9e75b8ab29bb4bdae6ee4bf44ed7f9e419)
    (cherry picked from commit 2113bf3ae9554b9c1cc3ab1cac9223b4022a56d2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/560022
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d5f8581e265a0227040488a1b565a0dc6faf9f13
Submitter: Zuul
Branch: stable/queens

commit d5f8581e265a0227040488a1b565a0dc6faf9f13
Author: Bob Fournier <email address hidden>
Date: Mon Mar 26 10:09:08 2018 -0400

    Remove config.json for old style nic config files when using new style

    This removes the config.json used by os-apply-config for old style nic
    config files in the script invoked for new style nic config files.
    This will avoid the case when the network files are set up by
    run-os-net-config.sh and then later overwritten by os-apply-config.

    Change-Id: I8f17a53c392aac657b00afdecbb5e58a2e192787
    Closes-Bug: 1758161
    (cherry picked from commit 9f4c691ccae4be6aa93e40450717b35f16a2a655)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 8.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.2 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/568604

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566336
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=08360e466730598e9ca5f270798cb3a510a5c9ee
Submitter: Zuul
Branch: master

commit 08360e466730598e9ca5f270798cb3a510a5c9ee
Author: mandreou <email address hidden>
Date: Fri May 4 17:30:11 2018 +0300

    Add .deployment.v1.deploy_on_servers to ffwd-upgrade prepare

    Before we update the heat stack for the ffwd-upgrade prepare we
    can use this to run any required commands on the overcloud.

    The current use case is removal of the
    /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json
    Otherwise after the upgrade prepare is executed, overcloud nodes
    will have empty config.json for os-net-config (also discussed [0])

    The current implementation of .deployment.v1.deploy_on_servers assumes
    all servers are managed by nova. This is OK for N..Q ffu but we will
    need to revisit in future, see [1] for more info.

    [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1561255
    [1]: https://bugs.launchpad.net/tripleo/+bug/1770356
    Related-Bug: 1758161
    Related:
        tripleo-common Ic138c4925c5c96d1cf718af7ae59bcf5f0ba96dc
    Change-Id: I77fdc9deab97c785725f09b8529c512950753f86

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/568604
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=12a5340c506157523df06888787aa8eff0cb69b8
Submitter: Zuul
Branch: stable/queens

commit 12a5340c506157523df06888787aa8eff0cb69b8
Author: mandreou <email address hidden>
Date: Fri May 4 17:30:11 2018 +0300

    Add .deployment.v1.deploy_on_servers to ffwd-upgrade prepare

    Before we update the heat stack for the ffwd-upgrade prepare we
    can use this to run any required commands on the overcloud.

    The current use case is removal of the
    /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json
    Otherwise after the upgrade prepare is executed, overcloud nodes
    will have empty config.json for os-net-config (also discussed [0])

    The current implementation of .deployment.v1.deploy_on_servers assumes
    all servers are managed by nova. This is OK for N..Q ffu but we will
    need to revisit in future, see [1] for more info.

    [0]: https://bugzilla.redhat.com/show_bug.cgi?id=1561255
    [1]: https://bugs.launchpad.net/tripleo/+bug/1770356
    Related-Bug: 1758161
    Depends-On: Ic138c4925c5c96d1cf718af7ae59bcf5f0ba96dc
    Change-Id: I77fdc9deab97c785725f09b8529c512950753f86
    (cherry picked from commit 0598141440882cc9a31e81626e7d01e0197eda80)