os-net-config fails on empty config file

[Lukas Bezdicka]

[root@overcloud-controller-0 ~]# os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes
[2017/02/20 02:25:21 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2017/02/20 02:25:21 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[2017/02/20 02:25:21 PM] [INFO] Ifcfg net config provider created.
Traceback (most recent call last):
  File "/bin/os-net-config", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 157, in main
    iface_array = yaml.load(cf.read()).get("network_config")
AttributeError: 'NoneType' object has no attribute 'get'

During minor update yum_update.sh always calls os-net-config unconditionally and sometimes fails on empty config.json file.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/436023

[Steven Hardy (shardy)]

> During minor update yum_update.sh always calls os-net-config unconditionally and sometimes fails on empty config.json file.

Can you expand on the situation where this will happen?

All deployments should specify some config I think, so although we can fix this so os-net-config treats an empty file as a no-op, I just want to ensure we're not working around some underlying issue in the deployment workflow or ordering?

[Steven Hardy (shardy)]

So it looks like this may be working around a potential race in the role templates:

https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/role.role.j2.yaml#L482

Note the UpdateDeployment has no depends_on, so it may run at the same time as the NetworkDeployment I think?

I suspect Lukas may be testing running a yum update during an overcloud create, and this is racing the NetworkDeployment creating the os-net-config configuration?

I'd be interested to see the results of adding a depends_on for all the UpdateDeployments (e.g depends_on: NetworkDeployment) as that may provide a more robust fix for this.

[Lukas Bezdicka (social-b)]

I tested after deployment, I also tested adding depends_on: NetworkDeployment and still /etc/os-net-config/config.json file is generated empty.

[Lukas Bezdicka (social-b)]

Similar check is being done in /usr/libexec/os-refresh-config/configure.d/20-os-net-config where it is

NET_CONFIG=$(os-apply-config --key os_net_config --type raw --key-default '')

if [ -n "$NET_CONFIG" ]; then

Maybe I should just change yum_update.sh to this.

[Steven Hardy (shardy)]

> NetworkDeployment and still /etc/os-net-config/config.json file is generated empty.

Can you share your network configuration environment/templates?

I'm not opposed to fixing this in os-net-config to tolerate and empty config, but I still don't understand why it is empty when we get to the update tasks.

[Steven Hardy (shardy)]

> Similar check is being done in /usr/libexec/os-refresh-config/configure.d/20-os-net-config

Note that approach is deprecated - all os-net-config execution for in-tree t-h-t network config templates is now done via this (which does contain a similar conditional)

https://github.com/openstack/tripleo-heat-templates/blob/master/network/scripts/run-os-net-config.sh#L112

So adding it to the yum update script is reasonable, or we could make os-net-config tolerate an empty config. I'd still like to understand why it's empty tho :)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/439504

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/439504
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=626b820b57498ff5002c5530962e6e4fd5644b51
Submitter: Jenkins
Branch: master

commit 626b820b57498ff5002c5530962e6e4fd5644b51
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/441202

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441202
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ff48624d03b48dc904bfae368f4be057c5dd2148
Submitter: Jenkins
Branch: stable/ocata

commit ff48624d03b48dc904bfae368f4be057c5dd2148
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227
    (cherry picked from commit 626b820b57498ff5002c5530962e6e4fd5644b51)

[Lukas Bezdicka (social-b)]

The changes didn't resolve the issue -
    "deploy_stderr": "[2017/03/13 01:28:22 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2017/03/13 01:28:22 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml\n[2017/03/13 01:28:22 PM] [INFO] Ifcfg net config provider created.\nTraceback (most recent call last):\n File \"/usr/bin/os-net-config\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 157, in main\n iface_array = yaml.load(cf.read()).get(\"network_config\")\nAttributeError: 'NoneType' object has no attribute 'get'\n",

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/444906

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lukas Bezdicka (<email address hidden>) on branch: master
Review: https://review.openstack.org/436023
Reason: https://review.openstack.org/444906

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/447426

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/444906
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b19d6306ea582dc31ebfd609475d9ac4e641e278
Submitter: Jenkins
Branch: master

commit b19d6306ea582dc31ebfd609475d9ac4e641e278
Author: Lukas Bezdicka <email address hidden>
Date: Mon Mar 13 14:39:12 2017 +0100

    Don't try to run os-net-config from yum_update.sh

    The UpdateDeployment already depends on NetworkDeployment.
    We should not run os-net-config unconditionally before update.

    Closes-Bug: #1666227

    Change-Id: I48cbf5de00d47c6fdad71ff24c00e9db05cec5d5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/447426
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5cd57aa2d342ca6851f25c65227183a54f16e008
Submitter: Jenkins
Branch: stable/ocata

commit 5cd57aa2d342ca6851f25c65227183a54f16e008
Author: Lukas Bezdicka <email address hidden>
Date: Mon Mar 13 14:39:12 2017 +0100

    Don't try to run os-net-config from yum_update.sh

    The UpdateDeployment already depends on NetworkDeployment.
    We should not run os-net-config unconditionally before update.

    Closes-Bug: #1666227

    Change-Id: I48cbf5de00d47c6fdad71ff24c00e9db05cec5d5
    (cherry picked from commit b19d6306ea582dc31ebfd609475d9ac4e641e278)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 6.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/513755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513755
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b7db8e853291874827bc25d17de06ad380b337a3
Submitter: Zuul
Branch: stable/newton

commit b7db8e853291874827bc25d17de06ad380b337a3
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227
    (cherry picked from commit 626b820b57498ff5002c5530962e6e4fd5644b51)
    (cherry picked from commit ff48624d03b48dc904bfae368f4be057c5dd2148)