STX-Openstack | Failing to application-upload - sysinv

[Gabriel Calixto de Paula]

Brief Description
-----------------
STX-openstack is failing when trying to do an application-upload with it.
Severity
--------
Critical
Steps to Reproduce
------------------
1- get the latest stx-openstack image
2- upload stx-openstack

Expected Behavior
------------------
The stx-openstack application should be uploaded

Actual Behavior
----------------
The stx-openstack application is not uploaded

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Two node system

Branch/Pull Time/Commit
-----------------------
STX - 20220926T230700Z

Last Pass
---------
Last sanity STX + STX-O 09/21/2022

Timestamp/Logs
--------------
sysinv.log:

28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app [-] 'int' object has no attribute 'lower': AttributeError: 'int' object has no attribute 'lower'2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app Traceback (most recent call last):2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2244, in perform_app_upload2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     self._save_images_list(app)2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 813, in _save_images_list2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     self._save_images_list_by_charts(app)2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 860, in _save_images_list_by_charts2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     chart_images = self._image.find_images_in_dict(y)2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4159, in find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     images = list(self._find_images_in_dict(var_dict))2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4146, in _find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     for result in self._find_images_in_dict(v):2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4146, in _find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     for result in self._find_images_in_dict(v):2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4146, in _find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     for result in self._find_images_in_dict(v):2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4146, in _find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     for result in self._find_images_in_dict(v):2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4119, in _find_images_in_dict2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app     if k.lower() == 'images':2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_app AttributeError: 'int' object has no attribute 'lower'2022-09-28 02:16:47.742 114871 ERROR sysinv.conductor.kube_appsysinv 2022-09-28 02:16:47.937 114871 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.sysinv 2022-09-28 02:16:47.937 114871 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: KubeAppUploadFailure: Upload of application stx-openstack (1.0-220-centos-stable-versioned) failed: 'int' object has no attribute 'lower'

Test Activity
-------------
Sanity

Workaround
----------
N/A

[Ghada Khalil (gkhalil)]

screening: stx.8.0 / high - issue impacting openstack sanity

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/config/+/859671

[Thales Elero Cervi (tcervi)]

It is surprisingly odd that an int is appearing on that part of the code, as a dict key.

Regardless of that, after ensuring the .lower() is only called for strings another issue came up:
sysinv 2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app [-] unhashable type: 'dict': TypeError: unhashable type: 'dict'
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app Traceback (most recent call last):
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 2246, in perform_app_upload
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app self._save_images_list(app)
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 817, in _save_images_list
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app images_to_download = self.get_image_tags_by_charts(app)
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 572, in get_image_tags_by_charts
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app app.sync_overrides_dir)
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 743, in _get_image_tags_by_charts_armada
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app download_imgs_list = self._image.generate_download_images_list(download_imgs, [])
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4243, in generate_download_images_list
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app self.generate_download_images_list(v, download_imgs_list)
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4243, in generate_download_images_list
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app self.generate_download_images_list(v, download_imgs_list)
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app File "/usr/lib64/python2.7/site-packages/sysinv/conductor/kube_app.py", line 4245, in generate_download_images_list
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app return list(set(download_imgs_list))
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app TypeError: unhashable type: 'dict'
2022-09-28 16:12:14.855 2171150 ERROR sysinv.conductor.kube_app
sysinv 2022-09-28 16:12:15.012 2171150 ERROR sysinv.conductor.kube_app [-] Application upload aborted!.

[Thales Elero Cervi (tcervi)]

Both issues seem to be related to this recent change on those methods: https://review.opendev.org/c/starlingx/config/+/858762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/859671
Committed: https://opendev.org/starlingx/config/commit/78040d2017c1edcf9f173af3e408499b23913529
Submitter: "Zuul (22348)"
Branch: master

commit 78040d2017c1edcf9f173af3e408499b23913529
Author: Thales Elero Cervi <email address hidden>
Date: Wed Sep 28 14:49:07 2022 -0300

    Add type checks to AppImageParser

    Recent changes [1] to AppImageParser _find_images_in_dict and
    generate_download_images_list methods made this code to break with both
    AttributeError and TypeError when stx-openstack application is being
    uploaded.

    This change includes extra protection against these types of errors and
    restablish the flow for generating stx-openstack image list based on its
    overrides.

    It also adds a new image resource to TestKubeAppImageParser unit tests,
    using an Openstack resource extracted from when debugging the original
    error. It should prevent this issue to happen again for future changes
    at AppImageParser logic.
    The original change to generate_download_images_list, for example, would
    fail the test:
        * TestKubeAppImageParser.test_generate_download_images_list

    [1] https://review.opendev.org/c/starlingx/config/+/858762

    Test Plan:
    PASS - Locally execute unit tests: TestKubeAppImageParser
    PASS - Build the sysinv package with this change
    PASS - Upload stx-openstack app
    PASS - Apply stx-openstack app

    Closes-Bug: 1991115

    Signed-off-by: Thales Elero Cervi <email address hidden>
    Change-Id: I8a1384bfefd12f8a893249853cbeae3a9d3661e0