Pike to Queens (master) undercloud upgrade failing in post config in _migrate_to_convergence

[Marios Andreou]

Just hit this on Pike containerized env on rdo cloud deployed with

        ## * Deploy the overcloud!
        ## ::
        openstack overcloud deploy \
            --templates /home/stack/tripleo-heat-templates \
            --libvirt-type qemu --control-flavor oooq_control --compute-flavor oooq_compute --ceph-storage-flavor oooq_ceph --block-storage-flavor oooq_blockstorage --swift-storage-flavor oooq_objectstorage --timeout 90 --ntp-server pool.ntp.org -e /home/stack/cloud-names.yaml -e /home/stack/neutronl3ha.yaml -e /home/stack/tripleo-heat-templates/environments/docker.yaml -e /home/stack/tripleo-heat-templates/environments/docker-ha.yaml -e /home/stack/containers-default-parameters.yaml -e /home/stack/tripleo-heat-templates/ci/environments/multinode-containers.yaml -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml -e /home/stack/network-environment.yaml -e /home/stack/overcloud_network_params.yaml -e /home/stack/resource-registry-nic-configs.yaml -e /home/stack/tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/stack/tripleo-heat-templates/environments/disable-telemetry.yaml --control-flavor baremetal --compute-flavor baremetal \

sudo yum install -y https://trunk.rdoproject.org/centos7/current/python2-tripleo-repos-0.0.1-0.20171116021457.15e17a8.el7.centos.noarch.rpm
sudo tripleo-repos current-tripleo-dev
sudo yum -y update python-tripleoclient
openstack undercloud upgrade
...

        2018-01-12 11:09:16,852 INFO: os-refresh-config completed successfully
        2018-01-12 11:09:26,568 INFO: Not creating flavor "baremetal" because it already exists.
        2018-01-12 11:09:26,670 INFO: Flavor baremetal updated to use custom resource class baremetal
        2018-01-12 11:09:26,670 INFO: Not creating flavor "control" because it already exists.
        2018-01-12 11:09:26,748 INFO: Flavor control updated to use custom resource class baremetal
        2018-01-12 11:09:26,748 INFO: Not creating flavor "compute" because it already exists.
        2018-01-12 11:09:26,961 INFO: Flavor compute updated to use custom resource class baremetal
        2018-01-12 11:09:26,961 INFO: Not creating flavor "ceph-storage" because it already exists.
        2018-01-12 11:09:27,045 INFO: Flavor ceph-storage updated to use custom resource class baremetal
        2018-01-12 11:09:27,045 INFO: Not creating flavor "block-storage" because it already exists.
        2018-01-12 11:09:27,117 INFO: Flavor block-storage updated to use custom resource class baremetal
        2018-01-12 11:09:27,117 INFO: Not creating flavor "swift-storage" because it already exists.
        2018-01-12 11:09:27,193 INFO: Flavor swift-storage updated to use custom resource class baremetal
        2018-01-12 11:09:27,203 INFO: Configuring Mistral workbooks
        2018-01-12 11:09:59,418 INFO: Mistral workbooks configured successfully
        2018-01-12 11:09:59,830 INFO: Not creating default plan "overcloud" because it already exists.
        2018-01-12 11:09:59,831 INFO: Configuring an hourly cron trigger for tripleo-ui logging
        2018-01-12 11:10:02,518 INFO: Added _member_ role to admin user
        2018-01-12 11:10:05,720 DEBUG: An exception occurred
        Traceback (most recent call last):
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1932, in install
            _post_config(instack_env, upgrade)
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1848, in _post_config
            _migrate_to_convergence(heat)
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1795, in _migrate_to_convergence
            for stack in heat.stacks.list():
          File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 136, in paginate
            stacks = self._list(url, 'stacks')
          File "/usr/lib/python2.7/site-packages/heatclient/common/base.py", line 114, in _list
            body = self.client.get(url).json()
          File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
            return self.request(url, 'GET', **kwargs)
          File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 317, in request
            raise exc.from_response(resp)
        HTTPInternalServerError: ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
        <html><head>
        <title>500 Internal Server Error</title>
        </head><body>
        <h1>Internal Server Error</h1>
        <p>The server encountered an internal error or
        misconfiguration and was unable to complete
        your request.</p>
        <p>Please contact the server administrator at
         [no address given] to inform them of the time this error occurred,
         and the actions you performed just before this error.</p>
        <p>More information about this error may be available
        in the server error log.</p>
        </body></html>

        2018-01-12 11:10:05,723 ERROR:
        #############################################################################
        Undercloud upgrade failed.

        Reason: ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
        <html><head>
        <title>500 Internal Server Error</title>
        </head><body>
        <h1>Internal Server Error</h1>
        <p>The server encountered an internal error or
        misconfiguration and was unable to complete
        your request.</p>
        <p>Please contact the server administrator at
         [no address given] to inform them of the time this error occurred,
         and the actions you performed just before this error.</p>
        <p>More information about this error may be available
        in the server error log.</p>
        </body></html>

        See the previous output for details about what went wrong. The full install
        log can be found at /home/stack/.instack/install-undercloud.log.

        #############################################################################

        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1932, in install
            _post_config(instack_env, upgrade)
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1848, in _post_config
            _migrate_to_convergence(heat)
          File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1795, in _migrate_to_convergence
            for stack in heat.stacks.list():
          File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 136, in paginate
            stacks = self._list(url, 'stacks')
          File "/usr/lib/python2.7/site-packages/heatclient/common/base.py", line 114, in _list
            body = self.client.get(url).json()
          File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
            return self.request(url, 'GET', **kwargs)
          File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 317, in request
            raise exc.from_response(resp)
        heatclient.exc.HTTPInternalServerError: ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
        <html><head>
        <title>500 Internal Server Error</title>
        </head><body>
        <h1>Internal Server Error</h1>
        <p>The server encountered an internal error or
        misconfiguration and was unable to complete
        your request.</p>
        <p>Please contact the server administrator at
         [no address given] to inform them of the time this error occurred,
         and the actions you performed just before this error.</p>
        <p>More information about this error may be available
        in the server error log.</p>
        </body></html>

        Command 'instack-upgrade-undercloud' returned non-zero exit status 1

[Marios Andreou (marios-b)]

note re-running the undercloud finished without errors on the second run.

[Pablo Iranzo Gómez (iranzo)]

Got this while performing OSP10 -> 11 -> 12 and last step at 13 of the undercloud:

openstack undercloud upgrade
Complementos cargados:product-id, search-disabled-repos, subscription-manager
No packages marked for update
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2472, in pre_upgrade
    for stack in heat.stacks.list():
  File "/usr/lib/python2.7/site-packages/heatclient/v1/stacks.py", line 136, in paginate
    stacks = self._list(url, 'stacks')
  File "/usr/lib/python2.7/site-packages/heatclient/common/base.py", line 114, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/heatclient/common/http.py", line 317, in request
    raise exc.from_response(resp)
heatclient.exc.HTTPInternalServerError: ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Command 'instack-pre-upgrade-undercloud' returned non-zero exit status 1