fm-rest-api helm chart failure to apply with error raising the pod

[Heitor Matsui]

Brief Description
-----------------
While trying to apply stx-openstack application, the apply fails during the fm-rest-api helm chart, with an error while trying to raise the fm-rest-api pod.

Severity
--------
Critical: System/Feature is not usable due to the defect

Steps to Reproduce
------------------
- system application-upload <stx-openstack.tgz_file>
- system application-apply stx-openstack

Expected Behavior
------------------
Application stx-openstack applied successfully

Actual Behavior
----------------
Application stx-openstack apply-failure with timeout

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
20211118T032155Z

Last Pass
---------
Unknown

Timestamp/Logs
--------------
2021-11-18T11:21:13.315200204Z stderr F ++ sed -nr 's/^sql_connection[[:space:]]?=[[:space:]]?//p' /etc/fm/fm.conf
2021-11-18T11:21:13.316056584Z stderr F + export 'SQL_CONNECTION=mysql+pymysql://fm:T*<email address hidden>:3306/fm'
2021-11-18T11:21:13.316071878Z stderr F + SQL_CONNECTION='mysql+pymysql://fm:T*<email address hidden>:3306/fm'
2021-11-18T11:21:13.316086819Z stderr F + echo 'mysql+pymysql://fm:T*<email address hidden>:3306/fm'
2021-11-18T11:21:13.316237372Z stderr F + python /usr/local/bin/fm_db_sync_event_suppression.py 'mysql+pymysql://fm:T*<email address hidden>:3306/fm'
2021-11-18T11:21:13.890444Z stderr F Traceback (most recent call last):
2021-11-18T11:21:13.890459948Z stderr F File "/usr/local/bin/fm_db_sync_event_suppression.py", line 28, in <module>
2021-11-18T11:21:13.890464438Z stderr F LOG = log.get_logger(__name__)
2021-11-18T11:21:13.89046758Z stderr F File "/usr/local/bin/fm_log.py", line 23, in get_logger
2021-11-18T11:21:13.890470774Z stderr F setup_logger(_loggers[name])
2021-11-18T11:21:13.890473396Z stderr F File "/usr/local/bin/fm_log.py", line 38, in setup_logger
2021-11-18T11:21:13.890481578Z stderr F facility=syslog_facility)
2021-11-18T11:21:13.890484707Z stderr F File "/usr/lib64/python2.7/logging/handlers.py", line 761, in __init__
2021-11-18T11:21:13.890531585Z stderr F self._connect_unixsocket(address)
2021-11-18T11:21:13.890536958Z stderr F File "/usr/lib64/python2.7/logging/handlers.py", line 789, in _connect_unixsocket
2021-11-18T11:21:13.890627864Z stderr F self.socket.connect(address)
2021-11-18T11:21:13.890642328Z stderr F File "/usr/lib64/python2.7/socket.py", line 224, in meth
2021-11-18T11:21:13.890650504Z stderr F return getattr(self._sock,name)(*args)
2021-11-18T11:21:13.890657253Z stderr F socket.error: [Errno 2] No such file or directory

Test Activity
-------------
Sanity

Workaround
----------
None

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/helm-charts/+/818454
Committed: https://opendev.org/starlingx/helm-charts/commit/4c6bd7423d776a914ef6743fcb50a6f7434d324a
Submitter: "Zuul (22348)"
Branch: master

commit 4c6bd7423d776a914ef6743fcb50a6f7434d324a
Author: Heitor Matsui <email address hidden>
Date: Thu Nov 18 16:14:06 2021 -0300

    Include CONTAINER env var in fm helm chart

    Currently the stx-openstack application apply is failing during
    the fm-rest-api helm chart, and the pod log indicate a file not
    found error while creating the log handler, as it uses a syslog
    handler (/dev/log socket) that doesn't exist on the pod.

    This change includes an environment variable to indicate that
    it is running in a container, so that when in this condition fm
    writes it's logs to standard output instead of using the syslog
    handler.

    Partial-Bug: 1951579
    Signed-off-by: Heitor Matsui <email address hidden>
    Change-Id: Icd853ae34851df8878843730143bb1e5f8d6a6fd

[Ghada Khalil (gkhalil)]

stx.6.0 / critical - this is causing a failure to install the stx-openstack application

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/utilities/+/818644

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/fault/+/818456
Committed: https://opendev.org/starlingx/fault/commit/f79b9a974895f794c802ec07d0dd6ffccf593be3
Submitter: "Zuul (22348)"
Branch: master

commit f79b9a974895f794c802ec07d0dd6ffccf593be3
Author: Heitor Matsui <email address hidden>
Date: Thu Nov 18 16:22:29 2021 -0300

    Use alternate log handler if running in container

    Currently the log is written to syslog, which in turn uses a
    socket /dev/log from the platform, but when running in a container
    such as in the stx-openstack application this log handler is not
    available.

    This commit uses an environment variable to be set on the
    fm-rest-api deployment definition to determine if it must the syslog
    handler, which is the case when running on the platform, or write the
    log to standard output, which is the case when running inside a
    container.

    Closes-Bug: 1951579
    Signed-off-by: Heitor Matsui <email address hidden>
    Change-Id: I3f2fe2953fb26ad1759f3bc10a358cb9d093192f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/utilities/+/818644
Committed: https://opendev.org/starlingx/utilities/commit/5d0b69af8437c68552a8ab90b46efb12d3bb2896
Submitter: "Zuul (22348)"
Branch: master

commit 5d0b69af8437c68552a8ab90b46efb12d3bb2896
Author: Thiago Brito <email address hidden>
Date: Fri Nov 19 18:41:26 2021 -0300

    Refactoring novaClient instantiation and unittests

    And adding infra for unittest execution.

    Partial-Bug: 1951579
    Signed-off-by: Thiago Brito <email address hidden>
    Change-Id: I58cefac9076db52333b41633bf2cbaa5441dc98c