rook-ceph-apps application apply aborted

[Weiyuan.Wang-Neu]

Brief Description
-----------------
rook-ceph-apps apply aborted.

Severity
--------
Major

Steps to Reproduce
------------------
1. attempted to apply rook-ceph-apps on Duplex and Standard Configuration with Controller Storage
modes.

2. system application-list: Application apply aborted.

Expected Behavior
------------------
application completed

Actual Behavior
----------------
sysinv.log shows:
sysinv 2021-07-13 06:51:02.163 103847 INFO sysinv.conductor.kube_app [-] All docker images for application rook-ceph-apps were successfully downloaded in 98 seconds
sysinv 2021-07-13 06:51:02.164 103847 INFO sysinv.conductor.kube_app [-] lifecycle hook for application rook-ceph-apps (1.0-8) started {'lifecycle_type': u'manifest', 'relative_timing': u'pre', 'mode': u'manual', 'operation': u'apply', 'extra': {}}.
sysinv 2021-07-13 06:51:02.172 103847 INFO sysinv.conductor.kube_app [-] lifecycle hook for application rook-ceph-apps (1.0-8) started {'lifecycle_type': u'armada-request', 'relative_timing': u'pre', 'operation': u'apply', 'extra': {}}.
sysinv 2021-07-13 06:51:03.018 103847 INFO sysinv.conductor.kube_app [-] Copy /opt/platform/armada/21.12/rook-ceph-apps to armada-api-775cff968d-gc6x7:/tmp/manifests .
sysinv 2021-07-13 06:51:03.173 103847 INFO sysinv.conductor.kube_app [-] Starting progress monitoring thread for app rook-ceph-apps
sysinv 2021-07-13 06:51:03.195 103847 INFO sysinv.conductor.kube_app [-] Copy /opt/platform/helm/21.12/rook-ceph-apps to armada-api-775cff968d-gc6x7:/tmp/overrides .
sysinv 2021-07-13 06:51:03.346 103847 INFO sysinv.conductor.kube_app [-] Armada apply command: 'armada apply --debug --enable-chart-cleanup /tmp/manifests/rook-ceph-apps/1.0-8/rook-ceph-apps-manifest.yaml --values /tmp/overrides/rook-ceph-apps/1.0-8/kube-system-rook-operator.yaml --values /tmp/overrides/rook-ceph-apps/1.0-8/kube-system-rook-ceph.yaml --values /tmp/overrides/rook-ceph-apps/1.0-8/kube-system-rook-ceph-provisioner.yaml '
sysinv 2021-07-13 06:51:04.220 103847 INFO sysinv.conductor.kube_app [-] processing chart: stx-rook-operator, overall completion: 33.0%
sysinv 2021-07-13 06:51:05.256 103847 ERROR sysinv.conductor.kube_app [-] Failed to apply application manifest /manifests/rook-ceph-apps/1.0-8/rook-ceph-apps-manifest.yaml with exit code 1. See /var/log/armada/rook-ceph-apps-apply_2021-07-13-06-51-02.log for details.
sysinv 2021-07-13 06:51:05.258 103847 INFO sysinv.conductor.kube_app [-] Exiting progress monitoring thread for app rook-ceph-apps
sysinv 2021-07-13 06:51:05.258 103847 INFO sysinv.conductor.kube_app [-] lifecycle hook for application rook-ceph-apps (1.0-8) started {'lifecycle_type': u'armada-request', 'relative_timing': u'post', 'operation': u'apply', 'extra': {'rc': False}}.
sysinv 2021-07-13 06:51:05.259 103847 INFO sysinv.conductor.kube_app [-] lifecycle hook for application rook-ceph-apps (1.0-8) started {'lifecycle_type': u'rbd', 'relative_timing': u'post', 'mode': u'manual', 'operation': u'apply', 'extra': {}}.
sysinv 2021-07-13 06:51:05.259 103847 INFO sysinv.conductor.kube_app [-] lifecycle hook for application rook-ceph-apps (1.0-8) started {'lifecycle_type': u'resource', 'relative_timing': u'post', 'mode': u'manual', 'operation': u'apply', 'extra': {}}.
sysinv 2021-07-13 06:51:05.447 103847 ERROR sysinv.conductor.kube_app [-] Application apply aborted!.

Reproducibility
---------------
<Reproducible/Intermittent/Seen once>
100% reproducible

System Configuration
--------------------
Duplex and Standard Configuration with Controller Storage

Branch/Pull Time/Commit
-----------------------
stx master as of
  20210629T220113Z
  20210705T232640Z
  20210710T020121Z.

Last Pass
---------
stx master as of 20210605T023117Z

Timestamp/Logs
--------------
2021-07-13 06:51:04.184 777 ERROR armada.handlers.tiller [-] [chart=kube-system-rook-operator]: Error while installing release stx-rook-operator: grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
 status = StatusCode.UNKNOWN
 details = "render error in "rook-operator/clusterrole.yaml": template: rook-operator/clusterrole.yaml:291:7: executing "rook-operator/clusterrole.yaml" at <((.Values.operator.agent) and .Values.operator.agent.mountSecurityMode) and ne .Values.operator.agent.mountSecurityMode "Any">: can't give argument to non-function (.Values.operator.agent) and .Values.operator.agent.mountSecurityMode"
 debug_error_string = "{"created":"@1626159064.183490393","description":"Error received from peer ipv4:127.0.0.1:24134","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"render error in "rook-operator/clusterrole.yaml": template: rook-operator/clusterrole.yaml:291:7: executing "rook-operator/clusterrole.yaml" at <((.Values.operator.agent) and .Values.operator.agent.mountSecurityMode) and ne .Values.operator.agent.mountSecurityMode "Any">: can't give argument to non-function (.Values.operator.agent) and .Values.operator.agent.mountSecurityMode","grpc_status":2}"

Test Activity
-------------

[chendongqi (chen-dq)]

This bug affects the feature test related to release 6.0 rook ceph

[chendongqi (chen-dq)]

20211023T032507Z (latest_green_build during testing) is not fixed

[chen haochuan (martin1982)]

rook-ceph/stx-rook-ceph/stx-rook-ceph/helm-charts/rook-operator/templates/clusterrole.yaml
remove these link, it will fix this issue

{{- if ((.Values.operator.agent) and .Values.operator.agent.mountSecurityMode) and ne .Values.operator.agent.mountSecurityMode "Any" }}
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: rook-ceph-agent-mount
 labels:
   operator: rook
   storage-backend: ceph
 aggregationRule:
   clusterRoleSelectors:
   - matchLabels:
    rbac.ceph.rook.io/aggregate-to-rook-ceph-agent-mount: "true"
    rules: []
--- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
 name: rook-ceph-agent-mount-rules
 labels:
 operator: rook
 storage-backend: ceph
 rbac.ceph.rook.io/aggregate-to-rook-ceph-agent-mount: "true"
 rules:
 - apiGroups:
 - "" resources:
 - secrets verbs:
 - get
{{- end }}

[chendongqi (chen-dq)]

Refer to the method described by chen haochuan (martin1982) to modify the file rook-ceph/stx-rook-ceph/stx-rook-ceph/helm-charts/rook-operator/templates/clusterrole.yaml, recompile source code to generate a new ISO, rook-ceph-apps application apply successfully, STX is deployed successfully,

[Ghada Khalil (gkhalil)]

screening: marking as low priority due to lack of activity

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/rook-ceph/+/818704

[chendongqi (chen-dq)]

check pick https://review.opendev.org/c/starlingx/rook-ceph/+/818704, manually compile and generate ISO, verify test rook ceph apply successfully

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Bob Church <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/rook-ceph/+/818704
Reason: Very old and no longer relevant for master branch content