[stable/queens] The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json

[Marios Andreou]

The tripleo-ci-centos-7-undercloud-upgrades is failing on queens for No such file or directory: '/etc/heat/policy.json - example at [1] and trace looks like:

  2018-10-08 01:12:42 | Traceback (most recent call last):
  2018-10-08 01:12:42 | File "<string>", line 1, in <module>
  2018-10-08 01:12:42 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2412, in pre_upgrade
  2018-10-08 01:12:42 | for stack in heat.stacks.list():
  2018-10-08 01:12:42 | File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 136, in paginate
  2018-10-08 01:12:42 | stacks = self._list(url, 'stacks')
  2018-10-08 01:12:42 | File "/usr/lib/python2.7/site-packages/heatclient/common/base.py", line 114, in _list
  2018-10-08 01:12:42 | body = self.client.get(url).json()
  2018-10-08 01:12:42 | File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
  2018-10-08 01:12:42 | return self.request(url, 'GET', **kwargs)
  2018-10-08 01:12:42 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 317, in request
  2018-10-08 01:12:42 | raise exc.from_response(resp)
  2018-10-08 01:12:42 | heatclient.exc.HTTPInternalServerError: ERROR: [Errno 2] No such file or directory: '/etc/heat/policy.json'
  2018-10-08 01:12:42 | Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1

[1]
http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/e4ad94e/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-08_01_12_42

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/608639

[Marios Andreou (marios-b)]

A check [0] was added for a deployed overcloud heat stack with https://review.openstack.org/#/c/570367/1 into the instack undercloud pre_upgrade function. However this is a problem for the undercloud jobs where there is no deployed overcloud.

We don't hit this on master, because we don't run the instack-undercloud code path there anymore. Tags were updated late last week so i suspect thats the reason we are now seeing it on queens (pike doesn't have this check at [2])

[0] https://github.com/openstack/instack-undercloud/blob/c220b84ef799960b3e4801940f5b5c77522840fa/instack_undercloud/undercloud.py#L2344-L2358
[1] https://github.com/openstack/python-tripleoclient/blob/71f51369a06ab0bf1ebc704ee3217b88bd069c01/tripleoclient/v1/undercloud.py#L159-L170
[2] https://github.com/openstack/instack-undercloud/blob/22320f51a4308398b0c4ab5bc04386d95c42424c/instack_undercloud/undercloud.py#L1937

[Rabi Mishra (rabi)]

As mentioned in the review, this is not related to that patch as it's failing to list the stacks in undercloud heat.

However from the heat api logs it's quite confusing.

It fails around 2018-10-08 01:12:42 and the corresponding heat api log

http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/e4ad94e/logs/undercloud/var/log/heat/heat_api.log.txt.gz#_2018-10-08_01_12_42_343

However, then it seems heat-api is restarted and it seems to work fine in a later time.

http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/e4ad94e/logs/undercloud/var/log/heat/heat_api.log.txt.gz#_2018-10-08_01_15_04_869

[Marios Andreou (marios-b)]

adding a note for now but something else must have fixed this the latest queens gate check https://review.openstack.org/#/c/567224 is green at [1] but previous run was not [2]

[1] http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/9938296/
[2] http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/15056cb/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz

 but latest OK tripleo-ci-centos-7-undercloud-upgrades SUCCESS in 1h 18m 31s

[Marios Andreou (marios-b)]

still green at http://logs.openstack.org/24/567224/117/check/tripleo-ci-centos-7-undercloud-upgrades/4ee7d6d/ queens gate check https://review.openstack.org/#/c/567224 so closing the bug for now we can re-open if it is seen again

[Alex Schultz (alex-schultz)]

Saw it in the gate again, http://logs.openstack.org/24/608324/1/check/tripleo-ci-centos-7-undercloud-upgrades/1d06342/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-10-10_13_16_08

[OpenStack Infra (hudson-openstack)]

Change abandoned by Marios Andreou (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/608639
Reason: invalid fix

[p.cipriano (ctpeter)]

Hi,

After running the openstack undercloud upgrade from pike to queens. We've encountered errors. Kindly see below logs for the errors.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2411, in pre_upgrade
    user_domain_name='Default')
  File "/usr/lib/python2.7/site-packages/os_client_config/__init__.py", line 86, in make_client
    return cloud.get_legacy_client(service_key, constructor)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 370, in get_legacy_client
    service_key, min_version=min_version, max_version=max_version)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 309, in get_session_endpoint
    endpoint = self._get_highest_endpoint(service_types, kwargs)
  File "/usr/lib/python2.7/site-packages/os_client_config/cloud_config.py", line 267, in _get_highest_endpoint
    return session.get_endpoint(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 942, in get_endpoint
    return auth.get_endpoint(self, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 379, in get_endpoint
    allow_version_hack=allow_version_hack, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 270, in get_endpoint_data
    service_catalog = self.get_access(session).service_catalog
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
    self.auth_ref = self.get_auth_ref(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 199, in get_auth_ref
    self._plugin = self._do_create_plugin(session)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 188, in _do_create_plugin
    'Cannot use v2 authentication with domain scope')
keystoneauth1.exceptions.discovery.DiscoveryFailure: Cannot use v2 authentication with domain scope
Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1
clean_up UpgradeUndercloud: Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1
END return value: 1

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

So, for the last error, it's as if we're using /v2 path[1] while we're using domain[2]

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/identity/generic/base.py#L150-L153

[2] https://github.com/openstack/instack-undercloud/blob/stable/queens/instack_undercloud/undercloud.py#L2410..L2411

Continuing digging.

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Oki, starting at comment we actually have a new bug, following it in https://bugs.launchpad.net/tripleo/+bug/1798553

[Alex Schultz (alex-schultz)]

This is failing in check now

[Rafael Folco (rafaelfolco)]

undercloud upgrades stable/queens still broken

http://logs.openstack.org/11/615911/2/check/tripleo-ci-centos-7-undercloud-upgrades/d69684b/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz

happening since 10/28

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_status:%20FAILURE%20AND%20message:%20%5C%22relabel%20failed%5C%22

[Janki Chhatbar (jankihchhatbar)]

I am seeing it in today's (8/11/2018) run as well.
http://logs.openstack.org/43/612243/5/check/tripleo-ci-centos-7-undercloud-upgrades/498f272/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz

[Martin Schuppert (mschuppert)]

Seeing this as well on stable/queens in

http://logs.openstack.org/84/618984/1/check/tripleo-ci-centos-7-undercloud-upgrades/2363463/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-11-20_12_20_05

[Sorin Sbarnea (ssbarnea)]

Is anyone working to fix this? As this blocks merges of other fixes.

[Alex Schultz (alex-schultz)]

So whatever is happening, the python-tripleoclient is not properly getting updated during the upgrade script execution.

http://logs.openstack.org/75/620975/1/check/tripleo-ci-centos-7-undercloud-upgrades/6d3151c/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz

2018-11-29 22:13:20 | Loaded plugins: fastestmirror, priorities
2018-11-29 22:13:20 | Loading mirror speeds from cached hostfile
2018-11-29 22:13:21 | 122 packages excluded due to repository priority protections
2018-11-29 22:13:22 | No packages marked for update
2018-11-29 22:13:24 | Loaded plugins: fastestmirror, priorities
2018-11-29 22:13:25 | Loading mirror speeds from cached hostfile
2018-11-29 22:13:26 | 122 packages excluded due to repository priority protections
2018-11-29 22:13:26 | No packages marked for update
2018-11-29 22:13:30 | Traceback (most recent call last):
2018-11-29 22:13:30 | File "<string>", line 1, in <module>
2018-11-29 22:13:30 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2410, in pre_upgrade
2018-11-29 22:13:30 | for stack in heat.stacks.list():
2018-11-29 22:13:30 | File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 136, in paginate
2018-11-29 22:13:30 | stacks = self._list(url, 'stacks')
2018-11-29 22:13:30 | File "/usr/lib/python2.7/site-packages/heatclient/common/base.py", line 114, in _list
2018-11-29 22:13:30 | body = self.client.get(url).json()
2018-11-29 22:13:30 | File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
2018-11-29 22:13:30 | return self.request(url, 'GET', **kwargs)
2018-11-29 22:13:30 | File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 317, in request
2018-11-29 22:13:30 | raise exc.from_response(resp)
2018-11-29 22:13:30 | heatclient.exc.HTTPInternalServerError: ERROR: [Errno 2] No such file or directory: '/etc/heat/policy.json'
2018-11-29 22:13:30 | Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1

The first yum update is a yum update to python-tripleoclient so if it's not getting updated, that means it's either not available for an update or was updated prior to when it was supposed to be. I'm still trying to figure out the ordering of what is happening because the reproducer is broken and if I manually run through the process it works fine.

[Alex Schultz (alex-schultz)]

Found it. So during the install-built-repo task the python-tripleoclient is being updated.

https://github.com/openstack/tripleo-quickstart-extras/blob/7fcb5c2801c93b2cf90f49195cccff8909753263/roles/install-built-repo/templates/install-built-repo.sh.j2#L34

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/621259

[Sorin Sbarnea (ssbarnea)]

Change is in gate queue now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/621259
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=05b5b272f38217ce09bc0e9a4a484f97bffcd2e2
Submitter: Zuul
Branch: master

commit 05b5b272f38217ce09bc0e9a4a484f97bffcd2e2
Author: Alex Schultz <email address hidden>
Date: Fri Nov 30 11:46:05 2018 -0700

    Add ability to disable yum update with gating repo

    Previously when the gating repo was enabled, we would blindly yum update
    with the repo. This probably shouldn't be the case as we should just be
    installing or updating the new packages after the repo has been
    installed. However to maintain backwards compatibility, a new variable
    called ib_gating_repo_update has been created with the default of True.
    To disable the yum update after isntalling built repo, set this to
    false.

    Change-Id: I5172134738b5b3d5146e45f1e0a7720b6e1d5a2c
    Closes-Bug: #1796639

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.