tripleo

Overcloud Deployment fails at a very late stage with "500 Internal Server Error"

Bug #1774895 reported by Ameen Ali on 2018-06-03

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Expired	Medium	Unassigned

Bug Description

Description
===========
I'm performing a TripleO Setup (Pike) on top of vMware ESXi (Containerized Overcloud). Setup, introspection, as well as initial steps of overcloud deployment (booting from PXE, loading the deploy images, rebooting with new hostnames, etc are working fine). At a later stage, the overcloud deployment exists with the following message:

After more debugging, I got the following detailed errors from stacks

Steps to reproduce
==================
* Install and prepare 4 VMs (undercloud, 2x Compute, 1x Controller) + 1 Router (VyOS). Kindly check Environment section below

* Perform Undercloud setup + Introspection of nodes successfully

* Launch overcloud deployment using the following command:
# openstack overcloud deploy --templates -e /home/stack/node-info.yaml -e /home/stack/local_overcloud_images -e /usr/share/openstack-tripleo-heat-templates/environments/docker.yaml

Expected result
===============
Overcloud Deployment to finish successfully

Actual result
=============
Overcloud deployment exits with the following messages:

2018-06-02 02:34:42Z [overcloud.AllNodesDeploySteps.ControllerHostPrepDeployment.0]: CREATE_COMPLETE state changed
2018-06-02 02:34:42Z [overcloud.AllNodesDeploySteps.ControllerHostPrepDeployment]: CREATE_COMPLETE Stack CREATE completed successfully
2018-06-02 02:34:43Z [overcloud.AllNodesDeploySteps.ControllerHostPrepDeployment]: CREATE_COMPLETE state changed
2018-06-02 02:34:43Z [overcloud.AllNodesDeploySteps.ControllerPreConfig]: CREATE_IN_PROGRESS state changed
2018-06-02 02:34:44Z [overcloud.AllNodesDeploySteps.ControllerPreConfig]: CREATE_COMPLETE state changed
2018-06-02 02:34:44Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-06-02 02:34:47Z [overcloud.AllNodesDeploySteps.ComputeDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-06-02 02:34:55Z [overcloud.AllNodesDeploySteps.ControllerDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-06-02 02:35:03Z [overcloud.AllNodesDeploySteps.BlockStorageDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-06-02 02:35:05Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step1]: CREATE_IN_PROGRESS state changed
2018-06-02 02:35:10Z [overcloud.AllNodesDeploySteps.ObjectStorageDeployment_Step1]: CREATE_COMPLETE state changed
2018-06-02 02:35:10Z [overcloud.AllNodesDeploySteps.BlockStorageDeployment_Step1]: CREATE_COMPLETE state changed
2018-06-02 02:35:10Z [overcloud.AllNodesDeploySteps.CephStorageDeployment_Step1]: CREATE_COMPLETE state changed
2018-06-02 02:35:22Z [0]: CREATE_IN_PROGRESS state changed
ERROR: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

Environment
===========
Openstack Pike
For more info about the overall network/compute setup, kindly refer to the first part of this article:
https://messeiry.com/deploying-redhat-openstack-rhos-12-pike-on-vmware-esxi/

Logs & Configs
==============

-- Content of local_overcloud_images are here: http://paste.openstack.org/show/722609/
-- Content of node-info.yaml are here: http://paste.openstack.org/show/722610/
-- Content of stack failures and more are here: http://paste.openstack.org/show/722611/

Revision history for this message

Ameen Ali (a-root-ameen) wrote on 2018-06-03:

Edit: -- Content of stack failures and more are here:
https://pastebin.com/yWR7t5RV

seems to be cut off on pate.openstack.org

Revision history for this message

Ameen Ali (a-root-ameen) wrote on 2018-06-04:

Download full text (11.8 KiB)

Update 01:

Running the deploy_steps_playbook.yaml directly on the failed compute node produces the following:

=================================================

(undercloud) [stack@director ~]$ openstack software deployment show 2ed6c2cc-3257-44b1-93ef-f46098b15491 --format value --column server_id
6530c389-54c8-4b54-9669-1ac2e7b6c497

(undercloud) [stack@director ~]$ openstack overcloud config download
The TripleO configuration has been successfully generated into: /home/stack/tripleo-6IXCCo-config

(undercloud) [stack@director tripleo-6IXCCo-config]$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory deploy_steps_playbook.yaml --limit overcloud-novacompute-0

PLAY [overcloud] ************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************
The authenticity of host '192.0.2.11 (192.0.2.11)' can't be established.
ECDSA key fingerprint is SHA256:ZQh57G38E/H8wdYKkMNIX9pzIAsZ+yaTQ68/hQYGEYg.
ECDSA key fingerprint is MD5:07:44:a4:7a:0a:bd:38:5f:12:b0:2f:ab:af:ba:75:5d.
Are you sure you want to continue connecting (yes/no)? ues
Please type 'yes' or 'no': yes
ok: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/aodh)
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/aodh-api)

TASK [aodh logs readme] *****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [ceilometer logs readme] ***********************************************************************************
skipping: [192.0.2.11]

TASK [Mount NFS on host] ****************************************************************************************
skipping: [192.0.2.11] => (item={u'NFS_OPTIONS': u'_netdev,bg,intr,context=system_u:object_r:glance_var_lib_t:s0', u'NFS_SHARE': u''})

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/glance)

TASK [glance logs readme] ***************************************************************************************
skipping: [192.0.2.11]

TASK [ensure ceph configurations exist] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/gnocchi)
skipping: [192.0.2.11] => (item=/var/log/containers/http...

Update 01:

Running the deploy_steps_playbook.yaml directly on the failed compute node produces the following:

=================================================

(undercloud) [stack@director ~]$ openstack software deployment show 2ed6c2cc-3257-44b1-93ef-f46098b15491 --format value --column server_id
6530c389-54c8-4b54-9669-1ac2e7b6c497

(undercloud) [stack@director ~]$ openstack overcloud config download
The TripleO configuration has been successfully generated into: /home/stack/tripleo-6IXCCo-config

(undercloud) [stack@director tripleo-6IXCCo-config]$  ansible-playbook -i /usr/bin/tripleo-ansible-inventory deploy_steps_playbook.yaml --limit overcloud-novacompute-0

PLAY [overcloud] ************************************************************************************************

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/aodh) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/aodh-api)

TASK [aodh logs readme] *****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [ceilometer logs readme] ***********************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/glance)

TASK [glance logs readme] ***************************************************************************************
skipping: [192.0.2.11]

TASK [ensure ceph configurations exist] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/gnocchi) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/gnocchi-api)

TASK [gnocchi logs readme] **************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/heat) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/heat-api)

TASK [heat logs readme] *****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/heat) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/heat-api-cfn)

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/horizon) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/horizon)

TASK [horizon logs readme] **************************************************************************************
skipping: [192.0.2.11]

TASK [stat /lib/systemd/system/iscsid.socket] *******************************************************************
skipping: [192.0.2.11]

TASK [Stop and disable iscsid.socket service] *******************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/keystone) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/keystone)

TASK [keystone logs readme] *************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [memcached logs readme] ************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent directories] ****************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/mysql) 
skipping: [192.0.2.11] => (item=/var/lib/mysql)

TASK [mysql logs readme] ****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/neutron) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/neutron-api)

TASK [neutron logs readme] **************************************************************************************
skipping: [192.0.2.11]

TASK [create /var/lib/neutron] **********************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/nova) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/nova-api)

TASK [nova logs readme] *****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11]

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/nova) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/nova-placement)

TASK [create persistent logs directory] *************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/panko) 
skipping: [192.0.2.11] => (item=/var/log/containers/httpd/panko-api)

TASK [panko logs readme] ****************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent directories] ****************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/rabbitmq) 
skipping: [192.0.2.11] => (item=/var/lib/rabbitmq)

TASK [rabbitmq logs readme] *************************************************************************************
skipping: [192.0.2.11]

TASK [create persistent directories] ****************************************************************************
skipping: [192.0.2.11] => (item=/var/log/containers/redis) 
skipping: [192.0.2.11] => (item=/var/run/redis)

TASK [redis logs readme] ****************************************************************************************
skipping: [192.0.2.11]

TASK [stat /var/log/containers/swift] ***************************************************************************
skipping: [192.0.2.11]

TASK [remove /var/log/containers/swift if it is a directory] ****************************************************
skipping: [192.0.2.11]

TASK [create persistent directories] ****************************************************************************
skipping: [192.0.2.11] => (item=/srv/node) 
skipping: [192.0.2.11] => (item=/var/log/swift)

TASK [Create swift logging symlink] *****************************************************************************
skipping: [192.0.2.11]

TASK [Format SwiftRawDisks] *************************************************************************************

TASK [Mount devices defined in SwiftRawDisks] *******************************************************************

TASK [create persistent logs directory] *************************************************************************
ok: [192.0.2.11]

TASK [ceilometer logs readme] ***********************************************************************************
fatal: [192.0.2.11]: FAILED! => {"msg": "Failed to get information on remote file (/var/log/ceilometer/readme.txt): Permission denied"}
...ignoring

TASK [create persistent logs directory] *************************************************************************
ok: [192.0.2.11]

TASK [neutron logs readme] **************************************************************************************
fatal: [192.0.2.11]: FAILED! => {"msg": "Failed to get information on remote file (/var/log/neutron/readme.txt): Permission denied"}
...ignoring

TASK [stat /lib/systemd/system/iscsid.socket] *******************************************************************
ok: [192.0.2.11]

TASK [Stop and disable iscsid.socket service] *******************************************************************
ok: [192.0.2.11]

TASK [create persistent directories] ****************************************************************************
ok: [192.0.2.11] => (item=/var/log/containers/nova)
ok: [192.0.2.11] => (item=/var/lib/nova)
ok: [192.0.2.11] => (item=/var/lib/libvirt)

TASK [nova logs readme] *****************************************************************************************
fatal: [192.0.2.11]: FAILED! => {"msg": "Failed to get information on remote file (/var/log/nova/readme.txt): Permission denied"}
...ignoring

TASK [ensure ceph configurations exist] *************************************************************************
ok: [192.0.2.11]

TASK [create libvirt persistent data directories] ***************************************************************
ok: [192.0.2.11] => (item=/etc/libvirt)
failed: [192.0.2.11] (item=/etc/libvirt/secrets) => {"changed": false, "item": "/etc/libvirt/secrets", "msg": "There was an issue creating /etc/libvirt/secrets as requested: [Errno 13] Permission denied: '/etc/libvirt/secrets'", "path": "/etc/libvirt/secrets", "state": "absent"}
failed: [192.0.2.11] (item=/etc/libvirt/qemu) => {"changed": false, "item": "/etc/libvirt/qemu", "msg": "There was an issue creating /etc/libvirt/qemu as requested: [Errno 13] Permission denied: '/etc/libvirt/qemu'", "path": "/etc/libvirt/qemu", "state": "absent"}
ok: [192.0.2.11] => (item=/var/lib/libvirt)
ok: [192.0.2.11] => (item=/var/log/containers/nova)
        to retry, use: --limit @/home/stack/tripleo-6IXCCo-config/deploy_steps_playbook.retry

PLAY RECAP ******************************************************************************************************
192.0.2.11                 : ok=10   changed=0    unreachable=0    failed=1

Emilien Macchi (emilienm) on 2018-06-04

Changed in tripleo:
milestone:	none → rocky-2
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

Ameen Ali (a-root-ameen) wrote on 2018-06-04:

Update 02:
By using -b, the playbook ran successfully. Also, I omit the --limit option to make it run on all nodes and it also finished successfully. However, I was not presented with output specifying overcloud endpoint. Am I supposed to re-run the (# openstack overcloud deploy) command?

Emilien Macchi (emilienm) on 2018-06-05

Changed in tripleo:
milestone:	rocky-2 → rocky-3

Emilien Macchi (emilienm) on 2018-07-26

Changed in tripleo:
milestone:	rocky-3 → rocky-rc1

Emilien Macchi (emilienm) on 2018-07-26

Changed in tripleo:
milestone:	rocky-rc1 → stein-1

Juan Antonio Osorio Robles (juan-osorio-robles) on 2018-10-30

Changed in tripleo:
milestone:	stein-1 → stein-2

Emilien Macchi (emilienm) on 2019-01-13

Changed in tripleo:
milestone:	stein-2 → stein-3

Alex Schultz (alex-schultz) on 2019-03-14

Changed in tripleo:
milestone:	stein-3 → train-1

Alex Schultz (alex-schultz) on 2019-06-07

Changed in tripleo:
milestone:	train-1 → train-2

Alex Schultz (alex-schultz) on 2019-07-29

Changed in tripleo:
milestone:	train-2 → train-3

Alex Schultz (alex-schultz) on 2019-09-11

Changed in tripleo:
milestone:	train-3 → ussuri-1

Emilien Macchi (emilienm) on 2019-12-19

Changed in tripleo:
milestone:	ussuri-1 → ussuri-2

wes hayutin (weshayutin) on 2020-02-10

Changed in tripleo:
milestone:	ussuri-2 → ussuri-3

wes hayutin (weshayutin) on 2020-04-07

Changed in tripleo:
status:	Triaged → Incomplete

wes hayutin (weshayutin) on 2020-04-13

Changed in tripleo:
milestone:	ussuri-3 → ussuri-rc3

wes hayutin (weshayutin) on 2020-05-26

Changed in tripleo:
milestone:	ussuri-rc3 → victoria-1

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

Marios Andreou (marios-b) on 2020-11-03

Changed in tripleo:
milestone:	victoria-3 → wallaby-1

Marios Andreou (marios-b) on 2020-12-08

Changed in tripleo:
milestone:	wallaby-1 → wallaby-2

Marios Andreou (marios-b) on 2021-01-29

Changed in tripleo:
milestone:	wallaby-2 → wallaby-3

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-02-17:

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone:	wallaby-3 → none

Revision history for this message

Launchpad Janitor (janitor) wrote on 2021-04-19:

[Expired for tripleo because there has been no activity for 60 days.]

Changed in tripleo:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.