fm-rest-api pod for openstack sometimes doesn't come up on AIO-SX

[Thiago Paiva Brito]

Brief Description
-----------------
Pods for fm-rest-api doesn't come up on AIO-SX

Severity
------------------------------
Major

Steps to Reproduce
-----------------------------
Apply stx-openstack

Expected Behavior
------------------
stx-openstack is applied succesfully

Actual Behavior
----------------
stx-openstack fail to apply. fm-rest-api pods are on CrashLoopBackoff.

Reproducibility
----------------------------
Intermittent

System Configuration
----------------------------
AIO-SX

Timestamp/Logs
----------------------------
[2021-02-11 08:33:45,542] 314 DEBUG MainThread ssh.send :: Send 'echo $?'
[2021-02-11 08:33:45,594] 436 DEBUG MainThread ssh.expect :: Output:
0
controller-0:~$
[2021-02-11 08:33:45,594] 69 INFO MainThread kube_helper.exec_kube_cmd:: exec_kube_cmd:kubectl get pod --all-namespaces --field-selector=status.phase=Running -o=wide | grep --color=never -v -E '([0-9])+/\1'
[2021-02-11 08:33:45,594] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2021-02-11 08:33:45,595] 314 DEBUG MainThread ssh.send :: Send 'kubectl get pod --all-namespaces --field-selector=status.phase=Running -o=wide | grep --color=never -v -E '([0-9])+/\1''
[2021-02-11 08:33:45,822] 436 DEBUG MainThread ssh.expect :: Output:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
openstack fm-rest-api-6fd5fc987b-pj7m7 0/1 CrashLoopBackOff 3 111m 172.16.192.111 controller-0
openstack horizon-5d49d6d5d5-8czkn 0/1 Running 0 2m52s 172.16.192.95 controller-0
openstack neutron-server-7c99cd7f68-kmstz 0/1 Running 2 118m 172.16.192.82 controller-0
openstack nova-compute-controller-0-937646f6-9g7kh 1/2 Running 2 118m 192.168.204.2 controller-0
openstack nova-conductor-c75f59f69-lnppg 0/1 Running 3 118m 172.16.192.101 controller-0

Test Activity
-------------
Sanity

[Thiago Paiva Brito (outbrito)]

Found this on fm-rest-api pod logs

$ kubectl logs fm-rest-api-557f4fb754-j77sh -n openstack
++ awk -F = '/sql_connection/{print $2}' /etc/fm/fm.conf
+ export 'SQL_CONNECTION= mysql+pymysql://fm:b0B'
+ SQL_CONNECTION=' mysql+pymysql://fm:b0B'
+ echo mysql+pymysql://fm:b0B
+ python /usr/local/bin/fm_db_sync_event_suppression.py mysql+pymysql://fm:b0B
Traceback (most recent call last):
  File "/usr/local/bin/fm_db_sync_event_suppression.py", line 96, in <module>
    engine = sqlalchemy.create_engine(postgresql_credentials)
  File "/var/lib/openstack/lib64/python2.7/site-packages/sqlalchemy/engine/__init__.py", line 456, in create_engine
    return strategy.create(*args, **kwargs)
  File "/var/lib/openstack/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 54, in create
    u = url.make_url(name_or_url)
  File "/var/lib/openstack/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 229, in make_url
    return _parse_rfc1738_args(name_or_url)
  File "/var/lib/openstack/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 288, in _parse_rfc1738_args
    return URL(name, **components)
  File "/var/lib/openstack/lib64/python2.7/site-packages/sqlalchemy/engine/url.py", line 71, in __init__
    self.port = int(port)
ValueError: invalid literal for int() with base 10: 'b0B'

[Thiago Paiva Brito (outbrito)]

The fm-rest-api problem is caused because oslo_db's password has a "=" character on it as we can see on the excerpt below:

| system_overrides | endpoints: |
| | identity: |
| | auth: |
| | admin: {password: ******, region_name: RegionOne, username: admin} |
| | fm: {password: ******, region_name: RegionOne} |
| | stx_admin: {password: ******, region_name: RegionOne, username: stx_admin} |
| | test: {password: ******, region_name: RegionOne, username: test} |
| | oslo_db: |
| | auth: |
| | admin: {password: ******} |
| | fm: {password: b0B=somethingelse} |
| | pod: |

But on https://opendev.org/starlingx/helm-charts/src/branch/master/fm-rest-api/fm-rest-api/helm-charts/fm-rest-api/templates/bin/_fm-rest-api.sh.tpl#L11, the startup script for fm-rest-api does an awk -F "=" on that line, after the yaml is mounted on fm.conf and therefore it breaks the line in 3, but only gets $2 (which is b0B).

[Thiago Paiva Brito (outbrito)]

Fix proposed: https://review.opendev.org/c/starlingx/helm-charts/+/776550

[Thiago Paiva Brito (outbrito)]

After https://review.opendev.org/c/starlingx/helm-charts/+/776550 was merged, checking out the results, I perceived that k8s in some cases adds spaces surrounding the "=" on fm.conf.

Proposing https://review.opendev.org/c/starlingx/helm-charts/+/778192 to fix that flaw.

[Ghada Khalil (gkhalil)]

stx.5.0 / high - armada issue affecting openstack sanity