armada-manifest upload failed when configuring bond with OVS container (vswitch-type=none)

[Chris Winnicki]

Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)

/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):

2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp

Severity
--------
Major

Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application

[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+

Expected Behavior
------------------
uploading stx-openstack should be successful

Actual Behavior
----------------
uploading stx-openstack results in failure

Reproducibility
---------------
100% (4 of 4)

System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):

Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"

Last Pass
---------
Never - first time execution
The same lab has no issue when using vswitch_type=ovs-dpdk

Timestamp/Logs
--------------
(Logs are attached)

Test Activity
-------------
System installation and configuration

Additional system config/topology:
----------------------------------

[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+

[Ghada Khalil (gkhalil)]

This appears to be an issue when using vswitch_type=none
The code appears to setup auto-bridge in helm/neutron.py
https://review.opendev.org/#/c/633924/
https://opendev.org/starlingx/config/commit/822b99c016c1f2bd0cb5236e46bfb6a55456bb3d

It may be best for this to be investigated by Forrest's team as they added support for this, but I will leave that decision between Yan and Forrest

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 release gating; cannot run the openstack application on systems w/ data AE running containerized ovs. This is a fairly standard config and should work.

[Yan Chen (ychen2u)]

Agree that Forrest team should take this issue. Thanks.

[cheng li (chengli3)]

openstack-helm community doesn't support adding bond ovs port and vlan ovs port[1] yet. In this bug it failed because of *ae* datanetwork interface.

[1] https://github.com/openstack/openstack-helm/blob/215c82f18c775a039bce4c107ada8fe05985b208/neutron/templates/bin/_neutron-linuxbridge-agent-init.sh.tpl

[cheng li (chengli3)]

I created the storyboard[1] for ovs bond support. The openstack-helm community is happy to see this feature implemented. I can go ahead to implement this feature.

[1] https://storyboard.openstack.org/#!/story/2005946

[Ghada Khalil (gkhalil)]

@Cheng, I think it would be great if you implement the feature in openstack-helm. However, given how close we are to the release date for stx.2.0, I am going to move this stx bug to stx.3.0. This will give us more time to pick up the new openstack-helm which includes bond support.

I reviewed this with Matt Peters (networking TL) and he agreed to the move to stx.3.0.

[cheng li (chengli3)]

The patch is in review
https://review.opendev.org/#/c/673950/

[Le, Huifeng (hle2)]

Move to stx4.0 due to external dependency on openstack-helm community can not meet stx3.0 schedule.

[Ghada Khalil (gkhalil)]

Assigning to Yong Hu as he took the action to re-assign this LP since Cheng Li is no longer contributing to the stx project.

[Ghada Khalil (gkhalil)]

Lowering the priority as there doesn't seem to be much progress on the upstream helm-openstack fix. We will not hold up stx.4.0 for this given that the issue was reported almost a year ago and is present in previous releases.