403 error in horizon log when try to update the flavor metadata (and admin user is logged out)

Bug #1821213 reported by Wendy Mitchell on 2019-03-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Medium
zhipeng liu

Bug Description

Brief Description
-----------------
STX: 403 error in horizon log when try to update the flavor metadata (and admin user is logged out)

Severity
--------
Major

Steps to Reproduce
------------------
1. Log into horizon as admin user
2. Open an existing flavor that has either no metadata set
or metadata set via the cli eg. hw:cpu_policy
3. Try to update metadata eg. add hw:cpu_policy dedicated (or just Save if the metadata was already there)

Expected Behavior
------------------
Click Save in the Update Flavor Metadata should have completed successfully in this case.

Actual Behavior
----------------
Error messages (and the admin user is logged out of horizon unexpectedly in handling the error)

128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET / HTTP/1.1" 302 - "-" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9019
128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9019 "http://172.16.0.85:80/" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "PATCH /api/nova/flavors/4967e8d1-1166-443d-936f-e4048b6628a6/extra-specs/ HTTP/1.1" 403 268
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "PATCH /api/nova/flavors/4967e8d1-1166-443d-936f-e4048b6628a6/extra-specs/ HTTP/1.1" 403 268 "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
2019-03-21 16:24:36.364736 Logging out user "admin".
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/logout/ HTTP/1.1" 302 -
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/logout/ HTTP/1.1" 302 - "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/login/ HTTP/1.1" 200 8955
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/login/ HTTP/1.1" 200 8955 "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 3217
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 3217 "http://128.224.151.54:31000/auth/login/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /header/ HTTP/1.1" 200 105
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /header/ HTTP/1.1" 200 105 "http://128.224.151.54:31000/auth/login/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET / HTTP/1.1" 302 -
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET / HTTP/1.1" 302 - "-" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9013
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9013 "http://172.16.0.85:80/" "kube-probe/1.12"

(Worth noting is that the get to metadefs succeeds but the post doesnt.)

Reproducibility
---------------
Reproducible

System Configuration
--------------------
2 node (any system)

Branch/Pull Time/Commit
-----------------------
Master as of date:
BUILD_DATE="2019-03-20 01:30:00 +0000

Timestamp/Logs
--------------
see inline

Ghada Khalil (gkhalil) wrote :

This is a horizon issue and will need to be reported in a horizon launchpad.

tags: added: stx.distro.openstack
Changed in starlingx:
importance: Undecided → Low
Ghada Khalil (gkhalil) wrote :

Based on severity, this issue is not deemed gating for stx.
However, the issue needs to be reported to the horizon project so that the community can consider addressing it.

Ghada Khalil (gkhalil) wrote :

From Tyler:
There's a good chance this is a result of improper configuration or policy settings in either horizon or nova, which i think would be our issue. I'm doubtful this is broken in upstream horizon as it's pretty basic functionality, i haven't really investigated yet though

Marking as release gating. As per Tyler's comment, this could be a config issue on the stx side.

Changed in starlingx:
importance: Low → Medium
tags: added: stx.2019.05 stx.gui
removed: stx.distro.openstack
Changed in starlingx:
status: New → Triaged
assignee: nobody → Tyler Smith (tyler.smith)
Ken Young (kenyis) on 2019-04-05
tags: added: stx.2.0
removed: stx.2019.05
Ghada Khalil (gkhalil) on 2019-04-08
tags: added: stx.retestneeded
Wendy Mitchell (wmitchellwr) wrote :

In horizon update metadata operations continue to fail
eg. Update flavor metadata
eg. Update image metadata

Load: 20190410T013000Z
2+2 (or any lab)
eg. Lab: IP_1_4

Wendy Mitchell (wmitchellwr) wrote :

In horizon update metadata operations continue to fail
eg. Update flavor metadata
eg. Update image metadata
eg. Edit image operation(s)

Load: 20190410T013000Z
2+2 (or any lab)
eg. Lab: IP_1_4

Bill Zvonar (billzvonar) wrote :

Assigned to Bruce for re-assignment.

Changed in starlingx:
assignee: Tyler Smith (tyler.smith) → Bruce Jones (brucej)
Bruce Jones (brucej) on 2019-06-21
Changed in starlingx:
assignee: Bruce Jones (brucej) → Cindy Xie (xxie1)
Yan Chen (ychen2u) on 2019-06-25
Changed in starlingx:
assignee: Cindy Xie (xxie1) → Yan Chen (ychen2u)
yong hu (yhu6) on 2019-07-08
tags: added: stx.distro.openstack
removed: stx.gui
yong hu (yhu6) wrote :

@Yan, do we have any update on this?

Yan Chen (ychen2u) wrote :

@yong, not yet received response from the comunity.

yong hu (yhu6) wrote :

@Yan what response did you expect from community? any background discussion on this issue?

Yan Chen (ychen2u) wrote :

Need community suggestion on how to config horizon to authorize metadata update/create/delete.
Asked in the community channel on slack, but no response yet.

yong hu (yhu6) on 2019-07-19
Changed in starlingx:
assignee: Yan Chen (ychen2u) → Shuquan Huang (shuquan)
yao (yaozhou) on 2019-07-22
Changed in starlingx:
assignee: Shuquan Huang (shuquan) → yao (yaozhou)
tags: added: stx.regression
yong hu (yhu6) wrote :

The extra property can be set by cmdline on 0724 cengn build, see the attachment.

I didn't test it on Horizon Web UI, because of the lack of GUI system at this moment.

But anyway, admin has at least a solution to change the flavor meta data.

yao (yaozhou) wrote :

On 0717 cengn build, I test it on both CLI and Horizon.
CLI can create or change metadata without porblem,
Horizon can reproduce the problem with 403 error.

yao (yaozhou) wrote :

I believe the reason of the bug has found:
The config file /etc/apache2/sites-enabled/000-default.conf in Horizon container, line 17

RewriteCond %{REQUEST_METHOD} !^(POST|PUT|GET|DELETE)

This configuration will forbidden all PATCH method, so not only flavor update metadata, all functions which uses PATCH method will be banned(volume update metadata...).

After change this configuration to:
RewriteCond %{REQUEST_METHOD} !^(POST|PUT|GET|DELETE|PATCH)
in horizon-etc and restart the pod and everything gets OK.

So in order to fix this, we have to change the deployment configuration.

yao (yaozhou) wrote :

I post the patch in openstack-helm:
https://review.opendev.org/#/c/673444/

Cindy Xie (xxie1) wrote :

the patch has been merged in upstream, what it takes to make the fix available in StarlingX build?

Ghada Khalil (gkhalil) wrote :
Changed in starlingx:
status: Triaged → In Progress
Ghada Khalil (gkhalil) wrote :

Since the commit for r/stx.2.0 was not merged in time for the release final compile on August 26, this bug is now re-gated to stx.3.0. The fix should be merged in master only. The review of r/stx.2.0 should be abandoned.

tags: added: stx.3.0
removed: stx.2.0

Change abandoned by Don Penney (<email address hidden>) on branch: r/stx.2.0
Review: https://review.opendev.org/678167
Reason: Abandoning per release management request

Change abandoned by YU CHENGDE (yu.chengde@99cloud.net) on branch: master
Review: https://review.opendev.org/678166
Reason: move to starlingx/openstack-armada-app
abandon the review

Change abandoned by Chris Friesen (<email address hidden>) on branch: master
Review: https://review.opendev.org/692071
Reason: This change is no longer necessary, the new version of openstack-helm introduced by https://review.opendev.org/#/c/683886 already includes the change being made here.

Chris Friesen (cbf123) wrote :

We upversioned openstack-helm in https://review.opendev.org/#/c/683886, and the new version includes the upstream patch merged at https://review.opendev.org/#/c/673444/

As such, we no longer need a local patch for this change.

Chris Friesen (cbf123) on 2019-11-13
Changed in starlingx:
status: In Progress → Fix Released
Wendy Mitchell (wmitchellwr) wrote :

Horizon errors reported when you attempt metadata update eg. flavor metadata
I would have expected this to be fixed in this recent load...

see screen grab attached
"20191120T023000Z"

[HW - supermicro-4]

Wendy Mitchell (wmitchellwr) wrote :
Changed in starlingx:
status: Fix Released → Confirmed
Tyler Smith (tyler.smith) wrote :

It may be fixed upstream but it looks like we still override the config with the old value not including PATCH in this commit: https://review.opendev.org/#/c/684166/

zhipeng liu (zhipengs) wrote :

Hi all,

Patch submitted due to it overriding the config.
https://review.opendev.org/#/c/696035/

Zhipeng

yong hu (yhu6) on 2019-11-27
Changed in starlingx:
status: Confirmed → In Progress
assignee: yao (yaozhou) → zhipeng liu (zhipengs)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers