Openstack APPLICATION UPLOAD GETTING FAILED - with LAG Data Interface

[ANIRUDH GUPTA]

Brief Description
-----------------
We are trying to setup dedicated Storage setup using StarlingX 4.0 over which we have certain observations/Error during the time of worker Node Configuration for the LAG type Data interfaces. Though we have tweaked the procedure a bit and successfully unlocked the Worker Nodes, but we are facing the error in upload the STX Application packages

Severity
--------

Critical

Steps to Reproduce
------------------

STEP 1:
We have created bonds for Data Network on Worker Node(reference document: https://docs.starlingx.io/configuration/host_interface_network_config.html#:~:text=When%20a%20host%20is%20added,system%20host%2Dif%2Dadd.)

     * system host-if-add worker-1 -m 1500 -a active_standby data1bond ae eth1 eth2
     * system host-if-add worker-1 -m 1500 -a active_standby data2bond ae eth3 eth4

STEP 2:
Then further as per the main documents when Configuring data interfaces for worker nodes ( reference document: https://docs.starlingx.io/deploy_install_guides/r4_release/bare_metal/dedicated_storage_install_kubernetes.html)

  * SPL=/tmp/tmp-system-port-list
  * SPIL=/tmp/tmp-system-host-if-list

Above files does not have the information of the created Bond interfaces for data Nodes,

Further in order to get the the value of ${DATA0IFUUID} and ${DATA1IFUUID} below mentioned commands are executed:

DATA0IF=<DATA-0-PORT>
DATA1IF=<DATA-1-PORT>
PHYSNET0='physnet0'
PHYSNET1='physnet1'
SPL=/tmp/tmp-system-port-list
SPIL=/tmp/tmp-system-host-if-list

# configure the datanetworks in sysinv, prior to referencing it
# in the ``system host-if-modify`` command'.
system datanetwork-add ${PHYSNET0} vlan
system datanetwork-add ${PHYSNET1} vlan

for NODE in worker-0 worker-1; do
  echo "Configuring interface for: $NODE"
  set -ex
  system host-port-list ${NODE} --nowrap > ${SPL}
  system host-if-list -a ${NODE} --nowrap > ${SPIL}
  DATA0PCIADDR=$(cat $SPL | grep $DATA0IF |awk '{print $8}')
  DATA1PCIADDR=$(cat $SPL | grep $DATA1IF |awk '{print $8}')
  DATA0PORTUUID=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $2}')
  DATA1PORTUUID=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $2}')
  DATA0PORTNAME=$(cat $SPL | grep ${DATA0PCIADDR} | awk '{print $4}')
  DATA1PORTNAME=$(cat $SPL | grep ${DATA1PCIADDR} | awk '{print $4}')
  DATA0IFUUID=$(cat $SPIL | awk -v DATA0PORTNAME=$DATA0PORTNAME '($12 ~ DATA0PORTNAME) {print $2}')
  DATA1IFUUID=$(cat $SPIL | awk -v DATA1PORTNAME=$DATA1PORTNAME '($12 ~ DATA1PORTNAME) {print $2}')
  system host-if-modify -m 1500 -n data0 -c data ${NODE} ${DATA0IFUUID}
  system host-if-modify -m 1500 -n data1 -c data ${NODE} ${DATA1IFUUID}
  system interface-datanetwork-assign ${NODE} ${DATA0IFUUID} ${PHYSNET0}
  system interface-datanetwork-assign ${NODE} ${DATA1IFUUID} ${PHYSNET1}
  set +ex
done

But in our case these values in the variable(${DATA0IFUUID} and ${DATA1IFUUID}) were not getting populated, therefore we have used the UUID received from command :
              System host-if-list worker-0
              System host-if-list worker-1

We observed that UUID of any ethernet interface mentioned in the file (SPIL=/tmp/tmp-system-host-if-list
) matches with the UUID obtained by running the above command(System host-if-list worker-0 ) , so as we did not get the UUID from the file in case LAG Bonds we passed the Values of UUID directly.

STEP 3:

Using the change, we were able to successfully unlock the worker Nodes.

[sysadmin at controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | unlocked | enabled | available |
| 3 | storage-0 | storage | unlocked | enabled | available |
| 4 | storage-1 | storage | unlocked | enabled | available |
| 5 | worker-0 | worker | unlocked | enabled | available |
| 6 | worker-1 | worker | unlocked | enabled | available |
+----+--------------+-------------+----------------+-------------+--------------+

STEP 4:
Then Further while running the application-upload:
              system application-upload stx-openstack-<version>-centos-stable-versioned.tgz

Expected Behavior
------------------
Openstack Application Tar should get Uploaded Successfully.

Actual Behavior
----------------

Error is being observed while uploading the Openstack Application Tar

2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/helm/helm.py", line 317, in _get_helm_chart_overrides
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app cnamespace))
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app File "/opt/platform/helm/20.06/stx-openstack/1.0-49-centos-stable-versioned/plugins/k8sapp_openstack/helm/neutron.py", line 45, in get_overrides
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app 'hosts': self._get_per_host_overrides()
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app File "/opt/platform/helm/20.06/stx-openstack/1.0-49-centos-stable-versioned/plugins/k8sapp_openstack/helm/neutron.py", line 105, in _get_per_host_overrides
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app 'auto_bridge_add': self._get_host_bridges(host)})
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app File "/opt/platform/helm/20.06/stx-openstack/1.0-49-centos-stable-versioned/plugins/k8sapp_openstack/helm/neutron.py", line 141, in _get_host_bridges
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app port_name = self._get_interface_port_name(iface)
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app File "/opt/platform/helm/20.06/stx-openstack/1.0-49-centos-stable-versioned/plugins/k8sapp_openstack/helm/neutron.py", line 280, in _get_interface_port_name
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app assert iface['iftype'] == constants.INTERFACE_TYPE_ETHERNET
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app AssertionError
2021-02-05 07:46:40.318 113059 ERROR sysinv.conductor.kube_app

Reproducibility
---------------

100% reproducible

Complete SYSINV Log file is attached in the mail.

[Ghada Khalil (gkhalil)]

screening: issue reported on stx.4.0; further investigation is required by the distro.openstack team

[zhipeng liu (zhipengs)]

Hi,

We have not implemented bond port feature.
See below links
https://bugs.launchpad.net/starlingx/+bug/1833463
https://storyboard.openstack.org/#!/story/2005946

Thanks!
Zhipeng

[Austin Sun (sunausti)]

currently this feature is not enabled, if needed, please discuss in TSC meeting.