_stackrc_check in instack_undercloud not working as intended

Bug #1663719 reported by Ben Nemec
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fix Released
Ben Nemec

Bug Description

The function is checking for the existence of /root/stackrc and ~/stackrc. The problem is that we don't have access to root's home directory as a regular user, so the check always fails. I'm not sure what impact that has on the existing logic that uses the function, but I want to reuse it for something else and the method is essentially useless as written.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (master)

Fix proposed to branch: master
Review: https://review.openstack.org/432435

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Ben Nemec (bnemec) wrote :

It turns out this has a pretty serious impact on the member role check, specifically in that it wasn't being run at all before, and seems to be failing when run in the ci undercloud upgrade job: http://logs.openstack.org/35/432435/1/check/gate-tripleo-ci-centos-7-undercloud-upgrades-nv/62bd182/logs/var/log/undercloud_upgrade.txt.gz

There's another serious issue with that logic: it assumes keystone is running at the start of the upgrade. This isn't the case because our upgrade process shuts down all the services before starting the upgrade. I'm not sure how to deal with that yet, unfortunately.

Note that this introduces the possibility of breaking the undercloud Heat on upgrade, at least based on my understanding of why the member role check exists.

Changed in tripleo:
importance: High → Critical
Revision history for this message
Ben Nemec (bnemec) wrote :
Download full text (3.6 KiB)

Okay, I can't figure out what's going on here. My mitaka undercloud has no _member_ role at all. My newton undercloud has a _member_ role, but it's not assigned to the admin user:

[centos@undercloud ~]$ openstack role assignment list --user admin --project admin
| Role | User | Project |
| 89202a5643a645359281fa395cd88de7 | aa6e698809014827bbd328068682d814 | 1a3b39eccc394ba3ae82caa907fe042c |
[centos@undercloud ~]$ openstack role list
| ID | Name |
| 3cb1fa14c18a47a6bb8ab85f75cb8423 | swiftoperator |
| 89202a5643a645359281fa395cd88de7 | admin |
| 8d26aa45539e4b8bb64b07342b06702d | heat_stack_user |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | ...


Revision history for this message
Ben Nemec (bnemec) wrote :

Okay, looking at the original bug (https://bugs.launchpad.net/tripleo/+bug/1571708) I see the change merged during the newton timeframe, but according to the downstream bug, this only affected people upgrading from a downstream-only kilo release to liberty+. Given that the workaround wasn't backported to any of the intervening releases, I'm not sure how effective it was anyway.

Based on what I'm seeing in the bz, I think we should just check for the _member_ role after deployment instead of before and add it to admin if it's present then. It takes that change out of puppet, which is not ideal, but at least it will work with the current upgrade process. And we're already having to interact with keystone directly from instack-undercloud either way.

Revision history for this message
Ben Nemec (bnemec) wrote :

Dropping the priority as there is a simple workaround for the problem this was intended to fix and users would have had to do that in the previous releases anyway.

I've updated the proposed patch to something that I believe should work, but if this doesn't make ocata then it's not the end of the world.

Changed in tripleo:
importance: Critical → High
Changed in tripleo:
milestone: ocata-rc1 → ocata-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/432435
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=9f6465fe8c6732da7bb4f401b230a17f021f47a8
Submitter: Jenkins
Branch: master

commit 9f6465fe8c6732da7bb4f401b230a17f021f47a8
Author: Ben Nemec <email address hidden>
Date: Fri Feb 10 19:27:13 2017 +0000

    Change _member_role_exists to work with current upgrade flow

    Because we now shut down all OpenStack services before upgrading
    the undercloud, we can't run API queries against them prior to
    running puppet. This means we now need to handle the _member_ role
    in a post config step.

    Note that this was only not failing before because the check in
    _stackrc_exists was always returning false, so the old code was
    never actually run. That also means it was not doing what it was
    supposed to, of course.

    Change-Id: Iaf81adc48321dbc985c50aeb5bfc0cc94810cb1d
    Closes-Bug: 1663719

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud

This issue was fixed in the openstack/instack-undercloud development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.