retry ovs-appctl on failure

[Bob Fournier]

Its possible that ovs-appctl may fail when setting the active slave on a bond if the primary interface is not fully up. The ovs-appctl call should be retried in order to allow time for the primary interface to come up. A failure should be logged if the problem persists.

For example, here ifup is run on ens27f0 and ens2sf1, the two interfaces in a bond, and the failure occurs when setting ens27f0 as the active slave:

Oct 11 13:52:05 overcloud-controller-0 os-collect-config: [2018/10/11 01:50:15 PM] [INFO] running ifup on interface: ens27f1
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: [2018/10/11 01:50:16 PM] [INFO] running ifup on interface: bond1
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: [2018/10/11 01:50:16 PM] [INFO] running ifup on interface: ens27f0
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: [2018/10/11 01:50:16 PM] [INFO] running ifup on interface: ens27f1
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: [2018/10/11 01:50:16 PM] [INFO] Running ovs-appctl bond/set-active-slave ('bond1', 'ens27f0')
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: Traceback (most recent call last):
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/bin/os-net-config", line 10, in <module>
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: sys.exit(main())
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/cli.py", line 270, in main
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: activate=not opts.no_activate)
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/impl_ifcfg.py", line 1149, in apply
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: self.bond_primary_ifaces[bond])
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 338, in ovs_appctl
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: self.execute(msg, '/bin/ovs-appctl', action, *parameters)
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/lib/python2.7/site-packages/os_net_config/__init__.py", line 288, in execute
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: processutils.execute(cmd, *args, **kwargs)
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 424, in execute
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: cmd=sanitized_cmd)
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: Command: /bin/ovs-appctl bond/set-active-slave bond1 ens27f0
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: Exit code: 2
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: Stdout: u''
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: Stderr: u'cannot make disabled slave active\novs-appctl: ovs-vswitchd: server returned an error\n'
Oct 11 13:52:05 overcloud-controller-0 os-collect-config: + RETVAL=1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/610714

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/610714
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=25c70ea0766cee5ed456d4acb94be90247ed45ca
Submitter: Zuul
Branch: master

commit 25c70ea0766cee5ed456d4acb94be90247ed45ca
Author: Bob Fournier <email address hidden>
Date: Mon Oct 15 15:36:01 2018 -0400

    Retry ovs-appctl command

    The ovs-appctl command may fail when setting an interface as a
    slave interface in a bond and the primary has not come up yet
    from a preceding ifup call. Retry the ovs-appctl and log an
    error if the command still fails.

    Change-Id: I3f04a5d6c3a6dc324a2978fdecd2a77df0bf1ba0
    Closes-Bug: 1797955

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/612460

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/612460
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=e78d45f3680aa68a328edaf58f7b7129935bbc59
Submitter: Zuul
Branch: stable/rocky

commit e78d45f3680aa68a328edaf58f7b7129935bbc59
Author: Bob Fournier <email address hidden>
Date: Mon Oct 15 15:36:01 2018 -0400

    Retry ovs-appctl command

    The ovs-appctl command may fail when setting an interface as a
    slave interface in a bond and the primary has not come up yet
    from a preceding ifup call. Retry the ovs-appctl and log an
    error if the command still fails.

    Change-Id: I3f04a5d6c3a6dc324a2978fdecd2a77df0bf1ba0
    Closes-Bug: 1797955
    (cherry picked from commit 25c70ea0766cee5ed456d4acb94be90247ed45ca)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/620135

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 10.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/620135
Committed: https://git.openstack.org/cgit/openstack/os-net-config/commit/?id=22ca42f79cea17912f76f3f4db3da3f130a0b6f4
Submitter: Zuul
Branch: stable/queens

commit 22ca42f79cea17912f76f3f4db3da3f130a0b6f4
Author: Bob Fournier <email address hidden>
Date: Mon Oct 15 15:36:01 2018 -0400

    Retry ovs-appctl command

    The ovs-appctl command may fail when setting an interface as a
    slave interface in a bond and the primary has not come up yet
    from a preceding ifup call. Retry the ovs-appctl and log an
    error if the command still fails.

    Change-Id: I3f04a5d6c3a6dc324a2978fdecd2a77df0bf1ba0
    Closes-Bug: 1797955
    (cherry picked from commit 25c70ea0766cee5ed456d4acb94be90247ed45ca)
    (cherry picked from commit e78d45f3680aa68a328edaf58f7b7129935bbc59)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 9.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-net-config 8.4.4 release.