os-net-config fails on empty config file

Bug #1666227 reported by Lukas Bezdicka on 2017-02-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Medium
Lukas Bezdicka

Bug Description

[root@overcloud-controller-0 ~]# os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes
[2017/02/20 02:25:21 PM] [INFO] Using config file at: /etc/os-net-config/config.json
[2017/02/20 02:25:21 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml
[2017/02/20 02:25:21 PM] [INFO] Ifcfg net config provider created.
Traceback (most recent call last):
  File "/bin/os-net-config", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 157, in main
    iface_array = yaml.load(cf.read()).get("network_config")
AttributeError: 'NoneType' object has no attribute 'get'

During minor update yum_update.sh always calls os-net-config unconditionally and sometimes fails on empty config.json file.

Fix proposed to branch: master
Review: https://review.openstack.org/436023

Changed in tripleo:
assignee: nobody → Lukas Bezdicka (social-b)
status: New → In Progress
Steven Hardy (shardy) wrote :

> During minor update yum_update.sh always calls os-net-config unconditionally and sometimes fails on empty config.json file.

Can you expand on the situation where this will happen?

All deployments should specify some config I think, so although we can fix this so os-net-config treats an empty file as a no-op, I just want to ensure we're not working around some underlying issue in the deployment workflow or ordering?

Changed in tripleo:
milestone: none → pike-1
importance: Undecided → Medium
Steven Hardy (shardy) wrote :

So it looks like this may be working around a potential race in the role templates:

https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/role.role.j2.yaml#L482

Note the UpdateDeployment has no depends_on, so it may run at the same time as the NetworkDeployment I think?

I suspect Lukas may be testing running a yum update during an overcloud create, and this is racing the NetworkDeployment creating the os-net-config configuration?

I'd be interested to see the results of adding a depends_on for all the UpdateDeployments (e.g depends_on: NetworkDeployment) as that may provide a more robust fix for this.

Lukas Bezdicka (social-b) wrote :

I tested after deployment, I also tested adding depends_on: NetworkDeployment and still /etc/os-net-config/config.json file is generated empty.

Lukas Bezdicka (social-b) wrote :

Similar check is being done in /usr/libexec/os-refresh-config/configure.d/20-os-net-config where it is

NET_CONFIG=$(os-apply-config --key os_net_config --type raw --key-default '')

if [ -n "$NET_CONFIG" ]; then

Maybe I should just change yum_update.sh to this.

Steven Hardy (shardy) wrote :

> NetworkDeployment and still /etc/os-net-config/config.json file is generated empty.

Can you share your network configuration environment/templates?

I'm not opposed to fixing this in os-net-config to tolerate and empty config, but I still don't understand why it is empty when we get to the update tasks.

Steven Hardy (shardy) wrote :

> Similar check is being done in /usr/libexec/os-refresh-config/configure.d/20-os-net-config

Note that approach is deprecated - all os-net-config execution for in-tree t-h-t network config templates is now done via this (which does contain a similar conditional)

https://github.com/openstack/tripleo-heat-templates/blob/master/network/scripts/run-os-net-config.sh#L112

So adding it to the yum update script is reasonable, or we could make os-net-config tolerate an empty config. I'd still like to understand why it's empty tho :)

Reviewed: https://review.openstack.org/439504
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=626b820b57498ff5002c5530962e6e4fd5644b51
Submitter: Jenkins
Branch: master

commit 626b820b57498ff5002c5530962e6e4fd5644b51
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227

Reviewed: https://review.openstack.org/441202
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ff48624d03b48dc904bfae368f4be057c5dd2148
Submitter: Jenkins
Branch: stable/ocata

commit ff48624d03b48dc904bfae368f4be057c5dd2148
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227
    (cherry picked from commit 626b820b57498ff5002c5530962e6e4fd5644b51)

tags: added: in-stable-ocata
Lukas Bezdicka (social-b) wrote :

The changes didn't resolve the issue -
    "deploy_stderr": "[2017/03/13 01:28:22 PM] [INFO] Using config file at: /etc/os-net-config/config.json\n[2017/03/13 01:28:22 PM] [INFO] Using mapping file at: /etc/os-net-config/mapping.yaml\n[2017/03/13 01:28:22 PM] [INFO] Ifcfg net config provider created.\nTraceback (most recent call last):\n File \"/usr/bin/os-net-config\", line 10, in <module>\n sys.exit(main())\n File \"/usr/lib/python2.7/site-packages/os_net_config/cli.py\", line 157, in main\n iface_array = yaml.load(cf.read()).get(\"network_config\")\nAttributeError: 'NoneType' object has no attribute 'get'\n",

Change abandoned by Lukas Bezdicka (<email address hidden>) on branch: master
Review: https://review.openstack.org/436023
Reason: https://review.openstack.org/444906

Reviewed: https://review.openstack.org/444906
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b19d6306ea582dc31ebfd609475d9ac4e641e278
Submitter: Jenkins
Branch: master

commit b19d6306ea582dc31ebfd609475d9ac4e641e278
Author: Lukas Bezdicka <email address hidden>
Date: Mon Mar 13 14:39:12 2017 +0100

    Don't try to run os-net-config from yum_update.sh

    The UpdateDeployment already depends on NetworkDeployment.
    We should not run os-net-config unconditionally before update.

    Closes-Bug: #1666227

    Change-Id: I48cbf5de00d47c6fdad71ff24c00e9db05cec5d5

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/447426
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5cd57aa2d342ca6851f25c65227183a54f16e008
Submitter: Jenkins
Branch: stable/ocata

commit 5cd57aa2d342ca6851f25c65227183a54f16e008
Author: Lukas Bezdicka <email address hidden>
Date: Mon Mar 13 14:39:12 2017 +0100

    Don't try to run os-net-config from yum_update.sh

    The UpdateDeployment already depends on NetworkDeployment.
    We should not run os-net-config unconditionally before update.

    Closes-Bug: #1666227

    Change-Id: I48cbf5de00d47c6fdad71ff24c00e9db05cec5d5
    (cherry picked from commit b19d6306ea582dc31ebfd609475d9ac4e641e278)

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b1 development milestone.

This issue was fixed in the openstack/tripleo-heat-templates 6.1.0 release.

Reviewed: https://review.openstack.org/513755
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b7db8e853291874827bc25d17de06ad380b337a3
Submitter: Zuul
Branch: stable/newton

commit b7db8e853291874827bc25d17de06ad380b337a3
Author: Steven Hardy <email address hidden>
Date: Wed Mar 1 09:51:20 2017 +0000

    Make UpdateDeployment depend on NetworkDeployment

    Prior to https://review.openstack.org/#/c/271450/ os-net-config was
    applied via os-refresh-config directly, which meant that even though
    UpdateDeployment and NetworkDeployment can be created concurrently,
    we'd always do the os-net-config step first.

    However now that we apply both steps via scripts (which are both handled
    via the same heat-config hook) we should add an explicit dependency to
    ensure the network is always fully configured before attempting to run
    any update. This should avoid the risk of e.g running an update on
    initial deployment before the network connectivity to access yum repos
    is in place.

    Change-Id: Idff7a95afe7b49b6384b1d0c78e76522fb1f8eb7
    Related-Bug: #1666227
    (cherry picked from commit 626b820b57498ff5002c5530962e6e4fd5644b51)
    (cherry picked from commit ff48624d03b48dc904bfae368f4be057c5dd2148)

tags: added: in-stable-newton
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers