Unexpected error adding tier with system storage-tier-add

[Wendy Mitchell]

Brief Description
-----------------
Unexpected error adding tier with system storage-tier-add
Permission denied: '/opt/platform/sysinv/19.09/crushmap_rule_update.bin'

Severity
--------
Major

Steps to Reproduce
------------------
From the cli, run the command to add a new storage tier

# system storage-tier-add ceph_cluster <newtiername>

Expected Behavior
-----------------
crush map missing the rule for the new gold tier

Actual Behavior
----------------
The command returns an unexpected error:

# system storage-tier-add ceph_cluster gold

Unexpected error while running command.
Command: ceph osd getcrushmap -o /opt/platform/sysinv/19.09/crushmap_rule_update.bin
Exit code: 1
Stdout: ''
Stderr: "Can't open output file /opt/platform/sysinv/19.09/crushmap_rule_update.bin: [Errno 13] Permission denied: '/opt/platform/sysinv/19.09/crushmap_rule_update.bin'\n"

Reproducibility
---------------
yes

System Configuration
--------------------
storage lab
lab: wcp 113-121

Branch/Pull Time/Commit
-----------------------
2019-11-20_20-00-00

Last Pass
---------

Timestamp/Logs
--------------
see
2019-11-22T17:46:22.000 controller-0 bash: info HISTORY: PID=3387542 UID=0 system storage-tier-add ceph_cluster gold

Test Activity
-------------
Regression

[Wendy Mitchell (wmitchellwr)]

Possible issue with the following commit
https://opendev.org/starlingx/config/commit/089cd3f420b7e9d5897fe4b811e4678432652ef4

[Bob Church (rchurch)]

Directory permissions for/opt/platform/sysinv/19.09 are as follows.

controller-0:~$ ls -al /opt/platform/sysinv
total 12
drwxr-xr-x 3 root root 4096 Nov 21 22:40 .
drwxr-xr-x 12 root root 4096 Nov 21 23:01 ..
drwxr-xr-x 2 root root 4096 Nov 22 14:46 19.09

The traceback shows the sysinv-api is executing this ceph command:

Command: ceph osd getcrushmap -o /opt/platform/sysinv/19.09/crushmap_rule_update.bin
Exit code: 1
Stdout: ''
Stderr: "Can't open output file /opt/platform/sysinv/19.09/crushmap_rule_update.bin: [Errno 13] Permission denied: '/opt/platform/sysinv/19.09/crushmap_rule_update.bin'\n"". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/storage_tier.py", line 257, in post
    new_tier = _create(self, tier)

  File "/usr/lib64/python2.7/site-packages/sysinv/api/controllers/v1/storage_tier.py", line 533, in _create
    self._ceph.crushmap_tiers_add()

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 407, in crushmap_tiers_add
    self._crushmap_add_tier(t)

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 390, in _crushmap_add_tier
    self._crushmap_rule_add(tier.name, replicate_by=replicate_by)

  File "/usr/lib64/python2.7/site-packages/sysinv/common/ceph.py", line 285, in _crushmap_rule_add
    stdout, __ = cutils.execute(*cmd, run_as_root=False)

  File "/usr/lib64/python2.7/site-packages/sysinv/common/utils.py", line 226, in execute
    cmd=' '.join(cmd))

ProcessExecutionError: Unexpected error while running command.
Command: ceph osd getcrushmap -o /opt/platform/sysinv/19.09/crushmap_rule_update.bin
Exit code: 1
Stdout: ''
Stderr: "Can't open output file /opt/platform/sysinv/19.09/crushmap_rule_update.bin: [Errno 13] Permission denied: '/opt/platform/sysinv/19.09/crushmap_rule_update.bin'\n"
: ProcessExecutionError: Unexpected error while running command.

With https://opendev.org/starlingx/config/commit/089cd3f420b7e9d5897fe4b811e4678432652ef4, the sysinv-api is no longer running with root group and this commit https://opendev.org/starlingx/ansible-playbooks/commit/dcdeea0254149233f1c0e5a6536a561cf7453cec has set permissions that will not allow the sysinv-api to write to this directory.

We need to do one of the following: change dir permissions or have the sysinv-api call the sysinv conductor to execute the command (the conductor has root permission).

[Ghada Khalil (gkhalil)]

stx.3.0 / medium priority - issue is recently introduced by other code changes as per above

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/696691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696691
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=05f770db5f53e9397c8e007195ece7193d5c7056
Submitter: Zuul
Branch: master

commit 05f770db5f53e9397c8e007195ece7193d5c7056
Author: Stefan Dinescu <email address hidden>
Date: Fri Nov 29 13:05:10 2019 +0200

    Set proper permission for /opt/platform/sysinv

    Based on reviews https://review.opendev.org/#/c/692439/2 and
    https://review.opendev.org/#/c/691714 the owner of the
    /opt/platform/sysinv folder was the "sysinv" user.

    However, in https://review.opendev.org/#/c/694774/1 the
    owner what changed to root:root. This cause the sysinv
    conductor processs to not be able to write to this folder
    as it does not have proper permissions.

    The solution is to change the folder owner back to sysinv.

    Change-Id: I74962ea69dfd6e9ea24e4fa51fc4e8bb9b6325ac
    Closes-bug: 1853642
    Signed-off-by: Stefan Dinescu <email address hidden>

[Ghada Khalil (gkhalil)]

The next step is to cherrypick to the r/stx.3.0 branch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/697492

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/697492
Committed: https://git.openstack.org/cgit/starlingx/ansible-playbooks/commit/?id=92b931077967741c4e0b2e3f1adfb24c008f0cbe
Submitter: Zuul
Branch: r/stx.3.0

commit 92b931077967741c4e0b2e3f1adfb24c008f0cbe
Author: Stefan Dinescu <email address hidden>
Date: Fri Nov 29 13:05:10 2019 +0200

    Set proper permission for /opt/platform/sysinv

    Based on reviews https://review.opendev.org/#/c/692439/2 and
    https://review.opendev.org/#/c/691714 the owner of the
    /opt/platform/sysinv folder was the "sysinv" user.

    However, in https://review.opendev.org/#/c/694774/1 the
    owner what changed to root:root. This cause the sysinv
    conductor processs to not be able to write to this folder
    as it does not have proper permissions.

    The solution is to change the folder owner back to sysinv.

    Change-Id: I74962ea69dfd6e9ea24e4fa51fc4e8bb9b6325ac
    Closes-bug: 1853642
    Signed-off-by: Stefan Dinescu <email address hidden>
    (cherry picked from commit 05f770db5f53e9397c8e007195ece7193d5c7056)

[Wendy Mitchell (wmitchellwr)]

verified
2019-12-10_20-00-00