helm-upload failed with permission denied on controller-1

[Peng Peng]

Brief Description
-----------------
application upload with .tgz fail, the upload status shows upload-failed. helm-upload shows adding file under ‘/www/pages/helm_charts/starlingx/hello-kitty.tgz’: Permission denied.

Severity
--------
Major

Steps to Reproduce
------------------
system application-upload -n hello-kitty -v 1.0 /home/sysadmin/custom_apps/hello-kitty.tgz'
helm-upload starlingx /home/sysadmin/custom_apps/hello-kitty.tgz'

TC-name: z_containers/test_custom_containers.py::test_launch_app_via_sysinv

Expected Behavior
------------------
uploaded success

Actual Behavior
----------------
upload-failed

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Multi-node system

Lab-name:
IP_1-4
WCP_71-75
WCP_113-121

Branch/Pull Time/Commit
-----------------------
stx master as of 20190620T013000Z

Last Pass
---------
20190613T013000Z

Timestamp/Logs
--------------
[2019-06-20 15:08:15,448] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-upload -n hello-kitty -v 1.0 /home/sysadmin/custom_apps/hello-kitty.tgz'
[2019-06-20 15:08:17,075] 387 DEBUG MainThread ssh.expect :: Output:
+---------------+----------------------------------+
| Property | Value |
+---------------+----------------------------------+
| active | False |
| app_version | 1.0 |
| created_at | 2019-06-20T15:08:16.955220+00:00 |
| manifest_file | manifest.yaml |
| manifest_name | hello-kitty |
| name | hello-kitty |
| progress | None |
| status | uploading |
| updated_at | None |
+---------------+----------------------------------+

[2019-06-20 15:08:43,954] 268 DEBUG MainThread ssh.send :: Send 'system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne application-list'
[2019-06-20 15:08:45,480] 387 DEBUG MainThread ssh.expect :: Output:
+---------------------+---------+-------------------------------+---------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| application | version | manifest name | manifest file | status | progress |
+---------------------+---------+-------------------------------+---------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------+
| hello-kitty | 1.0 | hello-kitty | manifest.yaml | upload-failed | Upload of application hello-kitty (1.0) failed: Command '['helm-upload', 'starlingx', u'/scratch/apps/hello-kitty/1.0/charts/hello-kitty. |
| | | | | | tgz']' returned non-zero exit status 1 |

[2019-06-20 15:09:56,587] 268 DEBUG MainThread ssh.send :: Send 'helm-upload starlingx /home/sysadmin/custom_apps/hello-kitty.tgz'
[2019-06-20 15:09:56,717] 387 DEBUG MainThread ssh.expect :: Output:
cp: cannot create regular file ‘/www/pages/helm_charts/starlingx/hello-kitty.tgz’: Permission denied
Problem adding /home/sysadmin/custom_apps/hello-kitty.tgz to helm chart registry.
controller-1:~$

Test Activity
-------------
Sanity

[Angie Wang (angiewang)]

The problem is that the ownerships of the helm repos are not right on the controller-1.
[sysadmin@controller-1 ~(keystone_admin)]$ ls -ltra /www/pages/helm_charts/
total 16
drwxr-xr-x 4 1877 root 4096 Jun 20 13:44 .
drwxr-xr-x 2 1877 root 4096 Jun 20 13:44 starlingx
drwxr-xr-x 2 1877 root 4096 Jun 20 14:00 stx-platform
drwxr-xr-x. 7 root root 4096 Jun 20 14:24 ..

The ownerships are correct on the controller-0.
controller-0:~$ ls -ltra /www/pages/helm_charts/
total 16
drwxr-xr-x 2 www root 4096 Jun 20 13:58 starlingx
drwxr-xr-x 4 www root 4096 Jun 20 13:58 .
drwxr-xr-x 2 www root 4096 Jun 20 13:58 stx-platform
drwxr-xr-x. 7 root root 4096 Jun 20 13:59 ..

Any application upload actions issued on controller-1 will fail due to the permission denied.
I didn't check further, not sure who/why the ownership was changed on controller-1.

[Ghada Khalil (gkhalil)]

So this issue only happens after a swact where controller-1 becomes active?

[Ghada Khalil (gkhalil)]

Marking as stx.2.0 gating; issue with permissions preventing application upload. Seems to be reproducible on multiple systems.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/668034

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/668035

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668034
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=f10485b4d5d9163875580c5f888b1bef5716c689
Submitter: Zuul
Branch: master

commit f10485b4d5d9163875580c5f888b1bef5716c689
Author: Angie Wang <email address hidden>
Date: Thu Jun 27 10:56:10 2019 -0400

    Fix the helm-upload failure on controller-1

    The user "www" is used to upload charts by using
    helm-upload cmd. The root cause for the helm-upload
    failure is the uid and gid of user "www" do not match
    the uid and gid on controller-0.

    Currently, www is created on controller-0 during ansible
    configuration and the uid is set to "1877" explicitly.
    On controller-1, it's created in puppet manifest but
    it doesn't set the uid.

    This commit is to add the user "www" in setup package so
    that it's created after each node installation and it
    makes sure the uid and gid are consistent on each node.

    Change-Id: I59f2b379eedc4edf206916798d9837f97917ef7a
    Closes-Bug: 1833603
    Signed-off-by: Angie Wang <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668035
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=a7d6d3115674854cdb0e73995d265cd33b444f16
Submitter: Zuul
Branch: master

commit a7d6d3115674854cdb0e73995d265cd33b444f16
Author: Angie Wang <email address hidden>
Date: Thu Jun 27 10:17:24 2019 -0400

    Fix the helm-upload failure on controller-1

    The user "www" is used to upload charts by using
    helm-upload cmd. The root cause for the helm-upload
    failure is the uid and gid of user "www" do not match
    the uid and gid on controller-0.

    Currently, www is created on controller-0 during ansible
    configuration and the uid is set to "1877" explicitly.
    On controller-1, it's created in puppet manifest but
    it doesn't set the uid.

    The user "www" should really be added in setup package so
    that it's created after each node installation and it makes
    sure the uid and gid are consistent on each node.
    This is done in review https://review.opendev.org/#/c/668034/

    However, we want to add the uid and gid in puppet as well
    to ensure they are identical on both controllers in case
    the user is not created during setup rpm installation.

    Change-Id: I54035739c15eb0f5254f00f5c22d44925316e81c
    Partial-Bug: 1833603
    Signed-off-by: Angie Wang <email address hidden>

[Peng Peng (ppeng)]

Verified on
Lab: WCP_63_66
Load: 20190708T233000Z