glance storage init pod throwing unaothorized error

[Gurpreet Singh]

output of openstack-swift-ks-user job:

++ openstack domain create --or-show --enable -f value -c id '--description=Domain for RegionOne/default' default
+ PROJECT_DOMAIN_ID=default
+ openstack domain show default
+-------------+--------------------+
| Field | Value |
+-------------+--------------------+
| description | The default domain |
| enabled | True |
| id | default |
| name | Default |
| tags | [] |
+-------------+--------------------+
+ USER_PROJECT_DESC='Service Project for RegionOne/default'
++ openstack project create --or-show --enable -f value -c id --domain=default '--description=Service Project for RegionOne/default' service
+ USER_PROJECT_ID=c27a05211aa84a8c82abec2c8a1fce4a
+ openstack project show c27a05211aa84a8c82abec2c8a1fce4a
+-------------+------------------------------------+
| Field | Value |
+-------------+------------------------------------+
| description | Service Project for RegionOne/default |
| domain_id | default |
| enabled | True |
| id | c27a05211aa84a8c82abec2c8a1fce4a |
| is_domain | False |
| name | service |
| parent_id | default |
| tags | [] |
+-------------+------------------------------------+
++ openstack domain create --or-show --enable -f value -c id '--description=Domain for RegionOne/default' default
+ USER_DOMAIN_ID=default
+ openstack domain show default
+-------------+--------------------+
| Field | Value |
+-------------+--------------------+
| description | The default domain |
| enabled | True |
| id | default |
| name | Default |
| tags | [] |
+-------------+--------------------+
+ USER_DESC='Service User for RegionOne/default/ceph'
++ openstack user create --or-show --enable -f value -c id --domain=default --project-domain=default --project=c27a05211aa84a8c82abec2c8a1fce4a '--description=Service User for RegionOne/default/ceph' --password=dummy swift
+ USER_ID=77302c5908594e9484c05d92b0d7d129
+ openstack user set --password=dummy 77302c5908594e9484c05d92b0d7d129
+ openstack user show 77302c5908594e9484c05d92b0d7d129
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| default_project_id | c27a05211aa84a8c82abec2c8a1fce4a |
| description | Service User for RegionOne/default/ceph |
| domain_id | default |
| enabled | True |
| id | 77302c5908594e9484c05d92b0d7d129 |
| name | swift |
| options | {} |
| password_expires_at | 2018-11-27T11:24:39.000000 |
+---------------------+--------------------------------------+
+ IFS=,
+ for SERVICE_OS_ROLE in '${SERVICE_OS_ROLES}'
+ ks_assign_user_role
++ openstack role create --or-show -f value -c id admin
+ USER_ROLE_ID=ca78e5b86f8b48218e2fa869d7fc7461
+ openstack role add --user=77302c5908594e9484c05d92b0d7d129 --user-domain=default --project-domain=default --project=c27a05211aa84a8c82abec2c8a1fce4a ca78e5b86f8b48218e2fa869d7fc7461
+ openstack role assignment list --role=ca78e5b86f8b48218e2fa869d7fc7461 --user-domain=default --user=77302c5908594e9484c05d92b0d7d129
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| ca78e5b86f8b48218e2fa869d7fc7461 | 77302c5908594e9484c05d92b0d7d129 | | c27a05211aa84a8c82abec2c8a1fce4a | | False |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
+ : member
++ openstack role create --or-show -f value -c id member
+ export USER_ROLE_ID=f940907661154385bbddb609d5d2fad7
+ USER_ROLE_ID=f940907661154385bbddb609d5d2fad7
+ ks_assign_user_role
++ openstack role create --or-show -f value -c id admin
+ USER_ROLE_ID=ca78e5b86f8b48218e2fa869d7fc7461
+ openstack role add --user=77302c5908594e9484c05d92b0d7d129 --user-domain=default --project-domain=default --project=c27a05211aa84a8c82abec2c8a1fce4a ca78e5b86f8b48218e2fa869d7fc7461
+ openstack role assignment list --role=ca78e5b86f8b48218e2fa869d7fc7461 --user-domain=default --user=77302c5908594e9484c05d92b0d7d129
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+
| ca78e5b86f8b48218e2fa869d7fc7461 | 77302c5908594e9484c05d92b0d7d129 | | c27a05211aa84a8c82abec2c8a1fce4a | | False |
+----------------------------------+----------------------------------+-------+----------------------------------+--------+-----------+

When glance storage init pods hit curl command to swift endpoint. It gives unauthorized. Below is the log:

kubectl logs -n openstack glance-storage-init-f4qds
+ '[' xswift == xrbd ']'
+ set -ex
+ '[' xswift == xpvc ']'
+ '[' xswift == xswift ']'
+ : internal
++ openstack token issue -f value -c id
+ OS_TOKEN=gAAAAABb_mx_QuCpXVQBoiyzGqXF_v1rPZ7TjoJ9RKoGp2hwMQVKXd3lsTHFbWWCiw-JNk_M9U29UKpzXavRISbxfz-MoofZo9WJOjISAvCgL58--121fLDFTL8pCoVFasNH71bZNczxOG1-A2lPz0MK4K_iHYxhJxAtVc4OkbSKvDcy2jbZCM0
++ openstack project show service -f value -c id
+ OS_PROJECT_ID=c27a05211aa84a8c82abec2c8a1fce4a
++ openstack endpoint list --service swift --interface internal -f value -c URL
++ awk -F '$' '{ print $1 }'
+ OS_SWIFT_ENDPOINT_PREFIX=http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_
+ OS_SWIFT_SCOPED_ENDPOINT=http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_c27a05211aa84a8c82abec2c8a1fce4a
+ curl --fail -i -X POST http://ceph-rgw.openstack.svc.cluster.local:8088/swift/v1/KEY_c27a05211aa84a8c82abec2c8a1fce4a -H 'X-Auth-Token: gAAAAABb_mx_QuCpXVQBoiyzGqXF_v1rPZ7TjoJ9RKoGp2hwMQVKXd3lsTHFbWWCiw-JNk_M9U29UKpzXavRISbxfz-MoofZo9WJOjISAvCgL58--121fLDFTL8pCoVFasNH71bZNczxOG1-A2lPz0MK4K_iHYxhJxAtVc4OkbSKvDcy2jbZCM0' -H 'X-Account-Meta-Temp-URL-Key: dummy'
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 401 Unauthorized

When I tried to generate token using swift credentials I see following issue:

The password is expired and needs to be changed for user: 77302c5908594e9484c05d92b0d7d129. (HTTP 401) (Request-ID: req-6195871f-b2e4-422b-808f-82400912ffeb)

[Gage Hugo (gagehugo)]

This looks like an invalid keystone setting, or system clock misconfiguration.