| 2019-06-19 19:39:29 |
Chris Winnicki |
bug |
|
|
added bug |
| 2019-06-19 19:39:29 |
Chris Winnicki |
attachment added |
|
cgcs-r430-3-4.wrs.com_ALL_NODES_20190619.tar https://bugs.launchpad.net/bugs/1833463/+attachment/5271615/+files/cgcs-r430-3-4.wrs.com_ALL_NODES_20190619.tar |
|
| 2019-06-20 23:37:22 |
Ghada Khalil |
summary |
stx-openstack armada-manifest upload-failedstx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config) |
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config) |
|
| 2019-06-21 00:42:36 |
Ghada Khalil |
tags |
|
stx.networking |
|
| 2019-06-21 03:09:00 |
Yan Chen |
starlingx: assignee |
|
Yan Chen (ychen2u) |
|
| 2019-06-21 12:45:55 |
Numan Waheed |
tags |
stx.networking |
stx.networking stx.retestneeded |
|
| 2019-06-21 14:02:08 |
Chris Winnicki |
description |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
20190604T144018Z
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
20190604T144018Z
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
|
| 2019-06-21 15:14:27 |
Ghada Khalil |
description |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
20190604T144018Z
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config). The system is being configured with AE data interfaces and vswitch_type=none
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
never - the same system works fine with ovs-dpdk.
This is the first time it's being configured with containerized ovs (vswitch_type=none)
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
|
| 2019-06-21 15:16:42 |
Ghada Khalil |
bug |
|
|
added subscriber Forrest Zhao |
| 2019-06-21 15:44:28 |
Ghada Khalil |
bug |
|
|
added subscriber cheng li |
| 2019-06-21 17:07:37 |
Chris Winnicki |
description |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config). The system is being configured with AE data interfaces and vswitch_type=none
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
never - the same system works fine with ovs-dpdk.
This is the first time it's being configured with containerized ovs (vswitch_type=none)
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
Never - first time execution
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
|
| 2019-06-21 17:26:38 |
Ghada Khalil |
description |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
Never - first time execution
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
Brief Description
-----------------
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config)
/var/log/sysinv.log (snippet of the exception during stx-openstack application-upload):
2019-06-19 19:01:06.319 109217 INFO sysinv.conductor.kube_app [-] Generating application overrides...
2019-06-19 19:01:06.833 109217 INFO sysinv.helm.neutron [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] _get_neutron_ml2_config={'ml2': {'physical_network_mtus': 'group0-data0:1500'}, 'ml2_type_flat': {'flat_networks': ''}}
2019-06-19 19:01:07.572 109217 ERROR sysinv.conductor.kube_app [-]
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app Traceback (most recent call last):
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1440, in perform_app_upload
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app self._save_images_list(app)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 469, in _save_images_list
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app armada_format=True, armada_chart_info=app.charts, combined=True)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 41, in _wrapper
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app return func(self, *args, **kwargs)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 470, in generate_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 257, in _get_helm_application_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 175, in _get_helm_chart_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app cnamespace))
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 45, in get_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 205, in _get_per_host_overrides
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 241, in _get_host_bridges
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/neutron.py", line 376, in _get_interface_port_name
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app AssertionError
2019-06-19 19:01:07.572 109217 TRACE sysinv.conductor.kube_app
2019-06-19 19:01:07.577 109217 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.
2019-06-19 19:01:07.577 109217 ERROR sysinv.openstack.common.rpc.amqp [req-fd397be7-dfc1-41e1-b31a-a6cd56b99a56 admin admin] Exception during message handling
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 438, in _process_data
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp **args)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 10710, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp self._app.perform_app_upload(rpc_app, tarfile)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 1457, in perform_app_upload
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp name=app.name, version=app.version, reason=e)
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp KubeAppUploadFailure: Upload of application stx-openstack (1.0-15-centos-stable-versioned) failed:
2019-06-19 19:01:07.577 109217 TRACE sysinv.openstack.common.rpc.amqp
Severity
--------
Major
Steps to Reproduce
------------------
1) Install 1+1 system as per system topology described below
2) Attempt to upload stx-openstack application
[sysadmin@controller-0 ~(keystone_admin)]$ system application-upload stx-openstack-1.0-15-centos-stable-versioned.tgz
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0-15-centos-stable-versioned |
| created_at | 2019-06-19T16:51:27.976976+00:00 |
| manifest_file | stx-openstack.yaml |
| manifest_name | armada-manifest |
| name | stx-openstack |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | uploading | validating and uploading charts |
+---------------------+--------------------------------+-------------------------------+--------------------+-----------+---------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system application-list
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
| platform-integ-apps | 1.0-7 | platform-integration-manifest | manifest.yaml | applied | completed |
| stx-openstack | 1.0-15-centos-stable-versioned | armada-manifest | stx-openstack.yaml | upload-failed | operation aborted, check logs for detail |
+---------------------+--------------------------------+-------------------------------+--------------------+---------------+------------------------------------------+
Expected Behavior
------------------
uploading stx-openstack should be successful
Actual Behavior
----------------
uploading stx-openstack results in failure
Reproducibility
---------------
100% (4 of 4)
System Configuration
--------------------
Two node system, IPv4 (refer to additional system info/topology below):
Branch/Pull Time/Commit
-----------------------
### StarlingX
### Built from master
###
OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190617T233000Z"
JOB="STX_build_master_master"
BUILD_BY="starlingx.build@cengn.ca"
BUILD_NUMBER="150"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-06-17 23:30:00 +0000"
Last Pass
---------
Never - first time execution
The same lab has no issue when using vswitch_type=ovs-dpdk
Timestamp/Logs
--------------
(Logs are attached)
Test Activity
-------------
System installation and configuration
Additional system config/topology:
----------------------------------
[sysadmin@controller-0 ~(keystone_admin)]$ system show
+----------------------+------------------------------------------------------+
| Property | Value |
+----------------------+------------------------------------------------------+
| contact | None |
| created_at | 2019-06-20T15:36:04.550715+00:00 |
| description | cgcs-r430-3-4.cumulus.wrs.com: setup by lab_setup.sh |
| https_enabled | True |
| location | None |
| name | cgcs-r430-3-4.cumulus.wrs.com |
| region_name | RegionOne |
| sdn_enabled | False |
| security_feature | spectre_meltdown_v1 |
| service_project_name | services |
| software_version | 19.01 |
| system_mode | duplex |
| system_type | All-in-one |
| timezone | UTC |
| updated_at | 2019-06-20T21:39:17.237300+00:00 |
| uuid | cfdbacf8-38d7-4feb-8bfe-ea691278e32a |
| vswitch_type | none |
+----------------------+------------------------------------------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 24edf111-a2b6-4226-89b0-3f5337fcf4cc | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| 307b9e93-ff0d-47c9-9e62-7c15e3690ce6 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| 38a11208-c9df-4de5-b96d-95ee2c623314 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 3cb15f88-3eee-4272-84d5-e16048538b84 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 968e8de0-e0a8-47fe-b8ae-4d294eab2118 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| b7580a4f-1055-4f6b-b6a8-50c7c0c68ee9 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| e86aaf36-9147-40c1-949e-e73802cef23f | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| e9f61841-7e31-42ec-b065-1695db03ac9f | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| f24c0cbb-df39-4ca9-a239-86ce05736dae | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'cluster0', u'mgmt0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system host-if-list controller-1 --nowrap
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
| 03070ae3-b1c7-43fb-b007-9e93806b66f6 | mgmt0 | platform | vlan | 157 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
| 470c34e7-ba9a-4171-bf32-0822ae334b85 | data0 | data | ae | None | [] | [u'enp5s0f0', u'enp5s0f1'] | [] | MTU=1500,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2,accelerated=True | [u'group0-data0'] |
| 4d993cf9-f8c8-44e0-a577-fbeb08997e63 | enp4s0f0 | None | ethernet | None | [u'enp4s0f0'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 576c50a9-d24f-478c-867d-1f34f993330d | pxeboot0 | platform | ae | None | [] | [u'enp4s0f0', u'enp4s0f1'] | [u'mgmt0', u'cluster0'] | MTU=9000,AE_MODE=802.3ad,AE_XMIT_POLICY=layer2 | [] |
| 7461d35d-ea47-4b5a-a1db-e8a7a4c7c680 | enp4s0f1 | None | ethernet | None | [u'enp4s0f1'] | [] | [u'pxeboot0'] | MTU=9000 | [] |
| 7e54a4df-e72f-48e0-aee1-a5e14a6fe5bf | enp5s0f1 | None | ethernet | None | [u'enp5s0f1'] | [] | [u'data0'] | MTU=1500 | [] |
| a9ed245d-4367-465d-8773-67a2792b5498 | oam0 | platform | ethernet | None | [u'eno1'] | [] | [] | MTU=1500 | [] |
| aa12a05a-a91d-4eae-bb98-21f387983854 | enp5s0f0 | None | ethernet | None | [u'enp5s0f0'] | [] | [u'data0'] | MTU=1500 | [] |
| b4693eaf-db80-4d0b-ae47-709548022f64 | cluster0 | platform | vlan | 158 | [] | [u'pxeboot0'] | [] | MTU=1500 | [] |
+--------------------------------------+----------+----------+----------+---------+---------------+----------------------------+-------------------------+-----------------------------------------------------------------+-------------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-0
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-0 | 018a9e6a-186b-4aef-b109-c14f91ff1fc2 | mgmt0 | mgmt |
| controller-0 | 11f641cc-bf0e-4041-9b34-b92cd6c7f058 | pxeboot0 | pxeboot |
| controller-0 | 823e7274-eb59-4499-bfb6-5a0965d9ce92 | oam0 | oam |
| controller-0 | fc1412cf-467d-4639-a5da-e57f3d018958 | cluster0 | cluster-host |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system interface-network-list controller-1
+--------------+--------------------------------------+----------+--------------+
| hostname | uuid | ifname | network_name |
+--------------+--------------------------------------+----------+--------------+
| controller-1 | 30b2bfe1-c1c6-4ddb-8192-cde50533e302 | mgmt0 | mgmt |
| controller-1 | 7f9af1cf-b5a8-41dc-9011-2659e4e0ef4a | cluster0 | cluster-host |
| controller-1 | d2685ba2-82d1-4e03-bfc3-a3e45925f985 | oam0 | oam |
| controller-1 | d88f6100-9206-44de-8703-cf515211d227 | pxeboot0 | pxeboot |
+--------------+--------------------------------------+----------+--------------+
[sysadmin@controller-0 ~(keystone_admin)]$ system datanetwork-list
+--------------------------------------+--------------+--------------+------+
| uuid | name | network_type | mtu |
+--------------------------------------+--------------+--------------+------+
| 5c1d363e-59d7-48e1-9174-db01e39e71e3 | group0-data0 | vlan | 1500 |
+--------------------------------------+--------------+--------------+------+ |
|
| 2019-06-21 19:00:44 |
Ghada Khalil |
tags |
stx.networking stx.retestneeded |
stx.2.0 stx.networking stx.retestneeded |
|
| 2019-06-21 19:01:10 |
Ghada Khalil |
starlingx: importance |
Undecided |
Medium |
|
| 2019-06-21 19:01:12 |
Ghada Khalil |
starlingx: status |
New |
Triaged |
|
| 2019-06-24 05:18:53 |
cheng li |
starlingx: assignee |
Yan Chen (ychen2u) |
cheng li (chengli3) |
|
| 2019-07-04 19:16:06 |
Ghada Khalil |
bug |
|
|
added subscriber Matt Peters |
| 2019-07-08 20:52:50 |
Ghada Khalil |
tags |
stx.2.0 stx.networking stx.retestneeded |
stx.3.0 stx.networking stx.retestneeded |
|
| 2019-07-18 20:30:30 |
Ghada Khalil |
summary |
stx-openstack armada-manifest upload-failed during Generating application overrides (_get_neutron_ml2_config) |
armada-manifest upload failed when configuring bond with OVS container (vswitch-type=none) |
|
| 2019-09-05 13:57:43 |
cheng li |
starlingx: status |
Triaged |
In Progress |
|
| 2019-09-10 17:35:25 |
Bill Zvonar |
bug |
|
|
added subscriber Bill Zvonar |
| 2019-09-11 12:19:31 |
Bill Zvonar |
removed subscriber Bill Zvonar |
|
|
|
| 2019-11-27 06:11:30 |
Le, Huifeng |
tags |
stx.3.0 stx.networking stx.retestneeded |
stx.4.0 stx.networking stx.retestneeded |
|
| 2020-03-19 13:47:32 |
Ghada Khalil |
starlingx: assignee |
cheng li (chengli3) |
yong hu (yhu6) |
|
| 2020-04-28 13:13:42 |
Bill Zvonar |
bug |
|
|
added subscriber Bill Zvonar |
| 2020-05-14 00:09:36 |
Bill Zvonar |
removed subscriber Bill Zvonar |
|
|
|
| 2020-05-26 02:42:56 |
Ghada Khalil |
tags |
stx.4.0 stx.networking stx.retestneeded |
stx.networking stx.retestneeded |
|
| 2020-05-26 02:43:02 |
Ghada Khalil |
starlingx: importance |
Medium |
Low |
|
