Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)

/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):

2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app     assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp     **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp     self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp     name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp


Severity
--------
Major

Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application

[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property      | Value                            |
+---------------+----------------------------------+
| active        | False                            |
| app_version   | 1.0-15-centos-stable-versioned   |
| created_at    | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml               |
| manifest_name | armada-manifest                  |
| name          | stx-openstack                    |
| progress      | None                             |
| status        | uploading                        |
| updated_at    | None                             |
+---------------+----------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list 
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application         | version                        | manifest name                 | manifest file      | status    | progress                        |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7                          | platform-integration-manifest | manifest.yaml      | applied   | completed                       |
| stx-openstack       | 1.0-15-centos-stable-versioned | armada-manifest               | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+

[sysadmin@controller-0 ~(keystone_admin)]$ system application-list 
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application         | version                        | manifest name                 | manifest file      | status        | progress                                 |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7                          | platform-integration-manifest | manifest.yaml      | applied       | completed                                |
| stx-openstack       | 1.0-15-centos-stable-versioned | armada-manifest               | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+



Expected Behavior
------------------
uploading stx-openstack should be successful

Actual Behavior
----------------
uploading stx-openstack results in failure


Reproducibility
---------------
100% (4 of 4)

System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):

Branch/Pull Time/Commit
-----------------------
### StarlingX
###     Built from master
###

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"


Last Pass
---------
20190604T144018Z

Timestamp/Logs
--------------
(Logs are attached)

Test Activity
-------------
System installation and configuration




Additional system config/topology:
----------------------------------

[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid                                 | name     | class    | type     | vlan id | ports         | uses i/f                   | used by i/f             | attributes                                                      | data networks     |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0     | platform | ethernet | None    | [u'eno1']     | []                         | []                      | MTU=1500                                                        | []                |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None     | ethernet | None    | [u'enp5s0f0'] | []                         | [u'data0']              | MTU=1500                                                        | []                |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None     | ethernet | None    | [u'enp4s0f1'] | []                         | [u'pxeboot0']           | MTU=9000                                                        | []                |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0    | data     | ae       | None    | []            | [u'enp5s0f0', u'enp5s0f1'] | []                      | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0    | platform | vlan     | 157     | []            | [u'pxeboot0']              | []                      | MTU=1500                                                        | []                |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None     | ethernet | None    | [u'enp4s0f0'] | []                         | [u'pxeboot0']           | MTU=9000                                                        | []                |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan     | 158     | []            | [u'pxeboot0']              | []                      | MTU=1500                                                        | []                |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None     | ethernet | None    | [u'enp5s0f1'] | []                         | [u'data0']              | MTU=1500                                                        | []                |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae       | None    | []            | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2                  | []                |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+


[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid                                 | name     | class    | type     | vlan id | ports         | uses i/f                   | used by i/f             | attributes                                                      | data networks     |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0    | platform | vlan     | 157     | []            | [u'pxeboot0']              | []                      | MTU=1500                                                        | []                |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0    | data     | ae       | None    | []            | [u'enp5s0f0', u'enp5s0f1'] | []                      | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None     | ethernet | None    | [u'enp4s0f0'] | []                         | [u'pxeboot0']           | MTU=9000                                                        | []                |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae       | None    | []            | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2                  | []                |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None     | ethernet | None    | [u'enp4s0f1'] | []                         | [u'pxeboot0']           | MTU=9000                                                        | []                |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None     | ethernet | None    | [u'enp5s0f1'] | []                         | [u'data0']              | MTU=1500                                                        | []                |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0     | platform | ethernet | None    | [u'eno1']     | []                         | []                      | MTU=1500                                                        | []                |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None     | ethernet | None    | [u'enp5s0f0'] | []                         | [u'data0']              | MTU=1500                                                        | []                |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan     | 158     | []            | [u'pxeboot0']              | []                      | MTU=1500                                                        | []                |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+


[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname     | uuid                                 | ifname   | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0    | mgmt         |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot      |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0     | oam          |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname     | uuid                                 | ifname   | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0    | mgmt         |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0     | oam          |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot      |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list 
+--------------------------------------+--------------+--------------+------+
| uuid                                 | name         | network_type | mtu  |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan         | 1500 |
+--------------------------------------+--------------+--------------+------+