403 error in horizon log when try to update the flavor metadata (and admin user is logged out)

[Wendy Mitchell]

Brief Description
-----------------
STX: 403 error in horizon log when try to update the flavor metadata (and admin user is logged out)

Severity
--------
Major

Steps to Reproduce
------------------
1. Log into horizon as admin user
2. Open an existing flavor that has either no metadata set
or metadata set via the cli eg. hw:cpu_policy
3. Try to update metadata eg. add hw:cpu_policy dedicated (or just Save if the metadata was already there)

Expected Behavior
------------------
Click Save in the Update Flavor Metadata should have completed successfully in this case.

Actual Behavior
----------------
Error messages (and the admin user is logged out of horizon unexpectedly in handling the error)

128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET / HTTP/1.1" 302 - "-" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9019
128.224.150.84 - - [21/Mar/2019:16:24:28 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9019 "http://172.16.0.85:80/" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "PATCH /api/nova/flavors/4967e8d1-1166-443d-936f-e4048b6628a6/extra-specs/ HTTP/1.1" 403 268
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "PATCH /api/nova/flavors/4967e8d1-1166-443d-936f-e4048b6628a6/extra-specs/ HTTP/1.1" 403 268 "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
2019-03-21 16:24:36.364736 Logging out user "admin".
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/logout/ HTTP/1.1" 302 -
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/logout/ HTTP/1.1" 302 - "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/login/ HTTP/1.1" 200 8955
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /auth/login/ HTTP/1.1" 200 8955 "http://128.224.151.54:31000/admin/flavors/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 3217
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /i18n/js/horizon+openstack_dashboard/ HTTP/1.1" 200 3217 "http://128.224.151.54:31000/auth/login/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /header/ HTTP/1.1" 200 105
128.224.150.84 - - [21/Mar/2019:16:24:36 +0000] "GET /header/ HTTP/1.1" 200 105 "http://128.224.151.54:31000/auth/login/" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Firefox/60.0"
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET / HTTP/1.1" 302 -
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET / HTTP/1.1" 302 - "-" "kube-probe/1.12"
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9013
128.224.150.84 - - [21/Mar/2019:16:24:38 +0000] "GET /auth/login/?next=/ HTTP/1.1" 200 9013 "http://172.16.0.85:80/" "kube-probe/1.12"

(Worth noting is that the get to metadefs succeeds but the post doesnt.)

Reproducibility
---------------
Reproducible

System Configuration
--------------------
2 node (any system)

Branch/Pull Time/Commit
-----------------------
Master as of date:
BUILD_DATE="2019-03-20 01:30:00 +0000

Timestamp/Logs
--------------
see inline

[Ghada Khalil (gkhalil)]

This is a horizon issue and will need to be reported in a horizon launchpad.

[Ghada Khalil (gkhalil)]

Based on severity, this issue is not deemed gating for stx.
However, the issue needs to be reported to the horizon project so that the community can consider addressing it.

[Ghada Khalil (gkhalil)]

From Tyler:
There's a good chance this is a result of improper configuration or policy settings in either horizon or nova, which i think would be our issue. I'm doubtful this is broken in upstream horizon as it's pretty basic functionality, i haven't really investigated yet though

Marking as release gating. As per Tyler's comment, this could be a config issue on the stx side.

[Wendy Mitchell (wmitchellwr)]

In horizon update metadata operations continue to fail
eg. Update flavor metadata
eg. Update image metadata

Load: 20190410T013000Z
2+2 (or any lab)
eg. Lab: IP_1_4

[Wendy Mitchell (wmitchellwr)]

In horizon update metadata operations continue to fail
eg. Update flavor metadata
eg. Update image metadata
eg. Edit image operation(s)

Load: 20190410T013000Z
2+2 (or any lab)
eg. Lab: IP_1_4

[Bill Zvonar (billzvonar)]

Assigned to Bruce for re-assignment.

[yong hu (yhu6)]

@Yan, do we have any update on this?

[Yan Chen (ychen2u)]

@yong, not yet received response from the comunity.

[yong hu (yhu6)]

@Yan what response did you expect from community? any background discussion on this issue?

[Yan Chen (ychen2u)]

Need community suggestion on how to config horizon to authorize metadata update/create/delete.
Asked in the community channel on slack, but no response yet.

[yong hu (yhu6)]

The extra property can be set by cmdline on 0724 cengn build, see the attachment.

I didn't test it on Horizon Web UI, because of the lack of GUI system at this moment.

But anyway, admin has at least a solution to change the flavor meta data.

[yao (yaozhou)]

On 0717 cengn build, I test it on both CLI and Horizon.
CLI can create or change metadata without porblem,
Horizon can reproduce the problem with 403 error.

[yao (yaozhou)]

I believe the reason of the bug has found:
The config file /etc/apache2/sites-enabled/000-default.conf in Horizon container, line 17

RewriteCond %{REQUEST_METHOD} !^(POST|PUT|GET|DELETE)

This configuration will forbidden all PATCH method, so not only flavor update metadata, all functions which uses PATCH method will be banned(volume update metadata...).

After change this configuration to:
RewriteCond %{REQUEST_METHOD} !^(POST|PUT|GET|DELETE|PATCH)
in horizon-etc and restart the pod and everything gets OK.

So in order to fix this, we have to change the deployment configuration.

[yao (yaozhou)]

I post the patch in openstack-helm:
https://review.opendev.org/#/c/673444/

[Cindy Xie (xxie1)]

the patch has been merged in upstream, what it takes to make the fix available in StarlingX build?

[Ghada Khalil (gkhalil)]

The equivalent patch needs to be applied to the starlingx/upstream repo
https://review.opendev.org/gitweb?p=starlingx/upstream.git;a=tree;f=openstack/openstack-helm;h=87344b5097cae1178e44d8448d11b4b716b91f9b;hb=HEAD

[Ghada Khalil (gkhalil)]

Reviews posted in starlingx
master: https://review.opendev.org/#/c/678166/
r/stx.2.0: https://review.opendev.org/#/c/678167/

[Ghada Khalil (gkhalil)]

Since the commit for r/stx.2.0 was not merged in time for the release final compile on August 26, this bug is now re-gated to stx.3.0. The fix should be merged in master only. The review of r/stx.2.0 should be abandoned.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Don Penney (<email address hidden>) on branch: r/stx.2.0
Review: https://review.opendev.org/678167
Reason: Abandoning per release management request

[OpenStack Infra (hudson-openstack)]

Change abandoned by YU CHENGDE ([e-mail]) on branch: master
Review: https://review.opendev.org/678166
Reason: move to starlingx/openstack-armada-app
abandon the review

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Friesen (<email address hidden>) on branch: master
Review: https://review.opendev.org/692071
Reason: This change is no longer necessary, the new version of openstack-helm introduced by https://review.opendev.org/#/c/683886 already includes the change being made here.

[Chris Friesen (cbf123)]

We upversioned openstack-helm in https://review.opendev.org/#/c/683886, and the new version includes the upstream patch merged at https://review.opendev.org/#/c/673444/

As such, we no longer need a local patch for this change.

[Wendy Mitchell (wmitchellwr)]

Horizon errors reported when you attempt metadata update eg. flavor metadata
I would have expected this to be fixed in this recent load...

see screen grab attached
"20191120T023000Z"

[HW - supermicro-4]

[Tyler Smith (tyler.smith)]

It may be fixed upstream but it looks like we still override the config with the old value not including PATCH in this commit: https://review.opendev.org/#/c/684166/

[zhipeng liu (zhipengs)]

Hi all,

Patch submitted due to it overriding the config.
https://review.opendev.org/#/c/696035/

Zhipeng

[Ghada Khalil (gkhalil)]

As per community review on 12/11, the recommendation is to cherry-pick this fix to r/stx.3.0 for the first maintenance release. Therefore, changing the priority to High (only high priority bugs are cherrypicked after the final compile).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696035
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=6af1446084817776b76b41bd366da3df2b3855e0
Submitter: Zuul
Branch: master

commit 6af1446084817776b76b41bd366da3df2b3855e0
Author: zhipengl <email address hidden>
Date: Wed Nov 27 01:45:38 2019 +0800

    Add 'PATCH' to request method in order to enable 'PATCH' method in Horizon

    Closes-Bug: #1821213

    Change-Id: Ic3d365e7bf9f725e4b39f3fa2df73a2c37bdd19b
    Signed-off-by: zhipengl <email address hidden>

[Wendy Mitchell (wmitchellwr)]

resolved
verified in 20200115T023003Z
add flavor, update metadata, delete flavor working

[Ghada Khalil (gkhalil)]

Zhipeng, This LP is marked as gating for stx.3.0. Please cherry-pick the code changes to the stx.3.0 branch if applicable or add a note explaining why it shouldn't be cherry-picked.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: r/stx.3.0
Review: https://review.opendev.org/723776

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/723776
Committed: https://git.openstack.org/cgit/starlingx/openstack-armada-app/commit/?id=e889f2516227ddb13f8ddb7bb294f184df59b6b9
Submitter: Zuul
Branch: r/stx.3.0

commit e889f2516227ddb13f8ddb7bb294f184df59b6b9
Author: zhipengl <email address hidden>
Date: Wed Nov 27 01:45:38 2019 +0800

    Add 'PATCH' to request method in order to enable 'PATCH' method in Horizon

    Closes-Bug: #1821213

    Change-Id: Ic3d365e7bf9f725e4b39f3fa2df73a2c37bdd19b
    Signed-off-by: zhipengl <email address hidden>
    (cherry picked from commit 6af1446084817776b76b41bd366da3df2b3855e0)