armada application-upload fails to parse yaml manifest

[Matt Peters]

Brief Description
-----------------
Armada application-upload causes yaml load error due to syntax error.

Severity
--------
Major

Steps to Reproduce
------------------
> system application-upload sample-app-1.0.tgz

Appears to be triggered by existing use of the PyYAML library that has not been updated to use ruamel.yaml library.

The following is triggering the error reported below, but there could be others that need to be udpated.
https://opendev.org/starlingx/config/src/branch/master/sysinv/sysinv/sysinv/sysinv/common/utils.py#L61

Expected Behavior
------------------
The application should be properly extracted and imported.

Actual Behavior
----------------
An internal error is generated and the application-upload fails (see log capture below).

Reproducibility
---------------
Reproducible when application manifest contains formatting not understood by PyYAML.

System Configuration
--------------------
All

Branch/Pull Time/Commit
-----------------------
Applications builds since this commit:
https://opendev.org/starlingx/root/commit/b2ff8cd529aee94b8ac5cd724448681298defc78

Last Pass
---------
Prior to the above commit.

Timestamp/Logs
--------------

sysinv 2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils [-] while scanning a plain scalar
  in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 19
found unexpected ':'
  in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 31
Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.: ScannerError: while scanning a plain scalar
  in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 19
found unexpected ':'
  in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 31
Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils Traceback (most recent call last):
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/sysinv/common/utils.py", line 1951, in find_manifest_file
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils mname, mfile = _is_manifest(yaml_file)
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/sysinv/common/utils.py", line 1936, in _is_manifest
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils for doc in docs:
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 83, in load_all
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils yield loader.get_data()
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/constructor.py", line 33, in get_data
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils return self.construct_document(self.get_node())
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/composer.py", line 27, in get_node

...snip...

2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 98, in check_event
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils self.current_event = self.state()
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 474, in parse_flow_sequence_first_entry
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils return self.parse_flow_sequence_entry(first=True)
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/parser.py", line 477, in parse_flow_sequence_entry
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils if not self.check_token(FlowSequenceEndToken):
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 116, in check_token
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils self.fetch_more_tokens()
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 252, in fetch_more_tokens
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils return self.fetch_plain()
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 680, in fetch_plain
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils self.tokens.append(self.scan_plain())
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils File "/usr/lib64/python2.7/site-packages/yaml/scanner.py", line 1309, in scan_plain
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils "Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.")
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils ScannerError: while scanning a plain scalar
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 19
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils found unexpected ':'
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils in "/scratch/sysinv-tmpdir/tmp9f0cTy/sample-app.yaml", line 849, column 31
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details.
2020-09-20 11:55:59.308 34090 ERROR sysinv.common.utils
sysinv 2020-09-20 11:55:59.351 34090 WARNING wsme.api [-] Client-side error: Application-upload rejected: manifest file is corrupted.: ClientSideError: Application-upload rejected: manifest file is corrupted.

Test Activity
-------------
Regression Testing

Workaround
----------
None.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/754500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/754501

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754501
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=e9fbe29f113fbbc89867fa951c98daf6dade9ac1
Submitter: Zuul
Branch: master

commit e9fbe29f113fbbc89867fa951c98daf6dade9ac1
Author: Yuxing Jiang <email address hidden>
Date: Sat Sep 26 09:53:34 2020 -0400

    Preserve quotes in building helm charts

    RoundTripLoader in ruamel.yaml will omit quotation marks. In some
    cases, the result by loading a yaml with RoundTripLoader and
    dumping it out will be unreadable by PYYAML. For example, the
    pattern:
      hosts: ["mon-logstash:port"]
    will be transformed to:
      hosts: [mon-logstash:port]
    Currently, the Armada project is using PYYAML. It's better to keep
    the quotes to allow the following process have to use PYYAML.

    Tested with rebuild the helm chart with the yaml has the pattern
    in the example, and the quotation marks are preserved.

    Change-Id: I7375a1ad498336baa337cca5af8f90028b753110
    Partial-Bug: 1896530
    Signed-off-by: Yuxing Jiang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754500
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=e4a13c34deae216de23cd0c5ba6c337da704a4d7
Submitter: Zuul
Branch: master

commit e4a13c34deae216de23cd0c5ba6c337da704a4d7
Author: Yuxing Jiang <email address hidden>
Date: Fri Sep 25 17:46:58 2020 -0400

    Preserve quotations when loading yaml using ruamel.yaml

    RoundTripLoader in ruamel.yaml will omit superfluous quotation
    marks. In some cases, the result by loading a yaml with and
    dumping it out will be unreadable by PyYAML. For example, the
    pattern:
      hosts: ["mon-logstash:port"]
    will be transformed to:
      hosts: [mon-logstash:port]
    Currently, the Armada project is using PyYAML. It's better to keep
    the quotes to allow the following process have to use PyYAML.

    This commit adds the parameter preserve_quotes=True to the
    RoundTripLoader to perserve the quotes when loading the original
    charts.

    Tested with a fresh install, upload the stx-monitor tarball with right
    formatted yaml files and successful apply the application.

    Change-Id: I1fd16b24bfb62ffc3bc885babbf6eb8a9356c421
    Depends-on: https://review.opendev.org/#/c/754501/
    Partial-Bug: 1896530
    Signed-off-by: Yuxing Jiang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/759762

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759762
Committed: https://git.openstack.org/cgit/starlingx/root/commit/?id=0636bbc87e8adf4e8faa61804f0ff222c8ef57b8
Submitter: Zuul
Branch: master

commit 0636bbc87e8adf4e8faa61804f0ff222c8ef57b8
Author: Yuxing Jiang <email address hidden>
Date: Mon Oct 26 10:27:06 2020 -0400

    Parse YAML with as YAML1.1 standard

    Ruamel.yaml is YAML1.2 compatible by default. The YAML1.2 does not
    support octals starts with 0 (the octals are required to start with
    0o). For example, the octal number 0755 with be transferred into 755.
    This commit set the ruamel.yaml load the filed as version 1.1. After
    this commit, the octal number will be transferred to decimal. For
    example, the octal 0755 will be re-write to 493. And it will be
    load/dump correctly in the following steps.

    Closes-Bug: 1896530
    Change-Id: If8be9d8d4b22ccce99307b5388313ba85a71be74
    Signed-off-by: Yuxing Jiang <email address hidden>