tripleo

keystone container failed to start in scenario000

Bug #1875352 reported by Amol Kahat on 2020-04-27

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Incomplete	High	Amol Kahat

Bug Description

Description
===========
Keystone container failed to start. Standalone deployment with scenario000 failed.

Job Description
===============

Job name: tripleo-podman-integration-centos-8-scenario000-standalone
Job build link: https://review.rdoproject.org/zuul/builds?job_name=tripleo-podman-integration-centos-8-scenario000-standalone

This job is not passed since long time.

Actual results
==============

TASK [tripleo_container_manage : Async container create/run] *******************
Monday 27 April 2020 11:37:33 +0000 (0:00:00.168) 0:05:59.870 **********
changed: [standalone] => (item=None)
changed: [standalone]

TASK [tripleo_container_manage : Check podman create status] *******************
Monday 27 April 2020 11:37:33 +0000 (0:00:00.434) 0:06:00.305 **********
FAILED - RETRYING: Check podman create status (60 retries left).
FAILED - RETRYING: Check podman create status (59 retries left).
FAILED - RETRYING: Check podman create status (58 retries left).
FAILED - RETRYING: Check podman create status (57 retries left).
FAILED - RETRYING: Check podman create status (56 retries left).
FAILED - RETRYING: Check podman create status (55 retries left).
FAILED - RETRYING: Check podman create status (54 retries left).
FAILED - RETRYING: Check podman create status (53 retries left).
FAILED - RETRYING: Check podman create status (52 retries left).
FAILED - RETRYING: Check podman create status (51 retries left).
FAILED - RETRYING: Check podman create status (50 retries left).
FAILED - RETRYING: Check podman create status (49 retries left).
FAILED - RETRYING: Check podman create status (48 retries left).
FAILED - RETRYING: Check podman create status (47 retries left).
FAILED - RETRYING: Check podman create status (46 retries left).
FAILED - RETRYING: Check podman create status (45 retries left).
FAILED - RETRYING: Check podman create status (44 retries left).
FAILED - RETRYING: Check podman create status (43 retries left).
FAILED - RETRYING: Check podman create status (42 retries left).
FAILED - RETRYING: Check podman create status (41 retries left).
FAILED - RETRYING: Check podman create status (40 retries left).
FAILED - RETRYING: Check podman create status (39 retries left).
FAILED - RETRYING: Check podman create status (38 retries left).
FAILED - RETRYING: Check podman create status (37 retries left).
FAILED - RETRYING: Check podman create status (36 retries left).
FAILED - RETRYING: Check podman create status (35 retries left).
FAILED - RETRYING: Check podman create status (34 retries left).
FAILED - RETRYING: Check podman create status (33 retries left).
FAILED - RETRYING: Check podman create status (32 retries left).
FAILED - RETRYING: Check podman create status (31 retries left).
FAILED - RETRYING: Check podman create status (30 retries left).
FAILED - RETRYING: Check podman create status (29 retries left).
FAILED - RETRYING: Check podman create status (28 retries left).
FAILED - RETRYING: Check podman create status (27 retries left).
FAILED - RETRYING: Check podman create status (26 retries left).
FAILED - RETRYING: Check podman create status (25 retries left).
FAILED - RETRYING: Check podman create status (24 retries left).
FAILED - RETRYING: Check podman create status (23 retries left).
FAILED - RETRYING: Check podman create status (22 retries left).
FAILED - RETRYING: Check podman create status (21 retries left).
FAILED - RETRYING: Check podman create status (20 retries left).
FAILED - RETRYING: Check podman create status (19 retries left).
FAILED - RETRYING: Check podman create status (18 retries left).
FAILED - RETRYING: Check podman create status (17 retries left).
FAILED - RETRYING: Check podman create status (16 retries left).
FAILED - RETRYING: Check podman create status (15 retries left).
FAILED - RETRYING: Check podman create status (14 retries left).
FAILED - RETRYING: Check podman create status (13 retries left).
FAILED - RETRYING: Check podman create status (12 retries left).
FAILED - RETRYING: Check podman create status (11 retries left).
FAILED - RETRYING: Check podman create status (10 retries left).
FAILED - RETRYING: Check podman create status (9 retries left).
FAILED - RETRYING: Check podman create status (8 retries left).
FAILED - RETRYING: Check podman create status (7 retries left).
FAILED - RETRYING: Check podman create status (6 retries left).
FAILED - RETRYING: Check podman create status (5 retries left).
FAILED - RETRYING: Check podman create status (4 retries left).
FAILED - RETRYING: Check podman create status (3 retries left).
FAILED - RETRYING: Check podman create status (2 retries left).
FAILED - RETRYING: Check podman create status (1 retries left).
ok: [standalone] => (item=None)
ok: [standalone]

TASK [tripleo_container_manage : Create facts for containers which changed or failed] ***
Monday 27 April 2020 11:42:44 +0000 (0:05:10.319) 0:11:10.624 **********
ok: [standalone]

TASK [tripleo_container_manage : Print the containers that failed to start] ****
Monday 27 April 2020 11:42:44 +0000 (0:00:00.112) 0:11:10.737 **********
fatal: [standalone]: FAILED! => changed=false
msg: '[''keystone_db_sync''] failed to start, check logs in /var/log/containers/stdouts/'

NO MORE HOSTS LEFT *************************************************************

PLAY RECAP *********************************************************************
standalone : ok=339 changed=140 unreachable=0 failed=1 skipped=154 rescued=0 ignored=0
undercloud : ok=21 changed=12 unreachable=0 failed=0 skipped=27 rescued=0 ignored=0
Monday 27 April 2020 11:42:44 +0000 (0:00:00.064) 0:11:10.802 **********
===============================================================================
tripleo_container_manage : Check podman create status ----------------- 310.32s
tripleo_container_image_prepare : Run tripleo_container_image_prepare logged to: /var/log/tripleo-container-image-prepare.log - 104.85s
tripleo_container_manage : Check podman create status ------------------ 16.03s
tripleo_container_manage : Wait for containers to be exited ------------ 10.74s
tripleo_container_manage : Check podman create status ------------------ 10.64s
Wait for puppet host configuration to finish ---------------------------- 9.78s
tripleo_firewall : Manage firewall rules -------------------------------- 8.08s
authorize httpd to listen on registry ports ----------------------------- 7.38s
Run NetworkConfig script ------------------------------------------------ 7.04s
Wait for puppet host configuration to finish ---------------------------- 6.66s
Wait for puppet host configuration to finish ---------------------------- 6.61s
tripleo_container_manage : Check podman create status ------------------- 6.23s
tripleo_container_manage : Check podman create status ------------------- 5.73s
tripleo_container_manage : Check podman create status ------------------- 5.70s
tripleo_container_manage : Check podman create status ------------------- 5.50s
tripleo_hieradata : Render hieradata from template ---------------------- 3.34s
tripleo_image_serve : ensure apache is installed ------------------------ 3.10s
tripleo_kernel : Set extra sysctl options ------------------------------- 2.67s
tripleo_clients_install : Run package install --------------------------- 2.25s
tripleo_bootstrap : Deploy required packages to bootstrap TripleO ------- 2.21s
Ansible execution failed. playbook: deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
Not cleaning ansible directory /home/zuul/standalone-ansible-h2cahqrr
Install artifact is located at /home/zuul/standalone-install-20200427114244.tar.bzip2

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Deployment Failed!

ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-djxd3rr6

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Tags:

Amol Kahat (amolkahat) on 2020-04-28

affects:

fuel → tripleo

Revision history for this message

Rabi Mishra (rabi) wrote on 2020-04-28:

Download full text (4.9 KiB)

Is there any successful run of the job earlier? Looks like some configuration issue or issue with images. mysql has not started for keystone_db_sync to happen.

/var/log/containers/keystone/keystone.log.txt.gz

2020-04-27 12:46:37.758 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -30 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:46:47.771 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -31 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:46:57.788 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -32 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:07.803 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -33 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:17.816 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -34 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:27.830 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -35 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:37.843 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -36 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:47.856 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -37 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:47:57.873 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -38 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:48:07.886 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -39 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:48:17.890 29 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -40 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.24.1' ([Errno 111] Connection refused)")
2020-04-27 12:48:27.904 29 WARNING oslo_...

I think it happens because pacemaker and libqb are different on the host vs inside the container.

host:

container:

()[mysql@e6473ba41b87 /]$ 
pacemaker-1.1.20-5.el7_7.2.x86_64
pacemaker-libs-1.1.20-5.el7_7.2.x86_64
pacemaker-remote-1.1.20-5.el7_7.2.x86_64
pacemaker-cli-1.1.20-5.el7_7.2.x86_64
puppet-pacemaker-0.8.1-0.20200203133231.83d23b3.el7.noarch
pacemaker-cluster-libs-1.1.20-5.el7_7.2.x86_64
libqb-1.0.1-7.el7.x86_64

and this results in:

https://logserver.rdoproject.org/19/26919/6/check/tripleo-ceph-integration-centos-7-scenario001-standalone-train/f326712/logs/undercloud/var/log/containers/stdouts/mysql_init_bundle.log.txt.gz

020-05-04T13:09:52.199888272+00:00 stderr F Error: /Stage[main]/Pacemaker::Stonith/Pacemaker::Property[Disable STONITH]/Pcmk_property[property--stonith-enabled]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200504-10-v9mh2h failed with code: 1 -> Error: unable to get cib
2020-05-04T13:09:52.199888272+00:00 stderr F . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-05-04T13:09:53.195928655+00:00 stderr F Error: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Property[galera-role-standalone]/Pcmk_property[property-standalone-galera-role]: Could not evaluate: backup_cib: Running: pcs cluster cib /var/lib/pacemaker/cib/puppet-cib-backup20200504-10-14wsb19 failed with code: 1 -> Error: unable to get cib
2020-05-04T13:09:53.195928655+00:00 stderr F . Either the cluster was not running or the versions of pcmk/pcs between host and container are not matching
2020-05-04T13:09:53.197083097+00:00 stdout F Notice: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Dependency Pcmk_property[property-standalone-galera-role] has failures: true
2020-05-04T13:09:53.197287929+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Bundle[galera-bundle]/Pcmk_bundle[galera-bundle]: Skipping because of failed dependencies
2020-05-04T13:09:53.198072623+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Pacemaker::Resource::Ocf[galera]/Pcmk_resource[galera]: Skipping because of failed dependencies
2020-05-04T13:09:53.198588812+00:00 stderr F Warning: /Stage[main]/Tripleo::Profile::Pacemaker::Database::Mysql_bundle/Exec[galera-ready]: Skipping because of failed dependencies
2020-05-04T13:09:53.213205898+00:00 stdout F Info: Creating state file /var/lib/puppet/state/state.yaml
2020-05-04T13:09:53.217818031+00:00 stderr F Error: Failed to apply catalog: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
2020-05-04T13:09:53.219131213+00:00 stdout F Changes:
2020-05-04T13:09:53.219131213+00:00 stdout F             Total: 6
2020-05-04T13:09:53.219131213+00:00 stdout F Events:
2020-05-04T13:09:53.219131213+00:00 stdout F           Failure: 2
2020-05-04T13:09:53.219131213+00:00 stdout F           Success: 6
2020-05-04T13:09:53.219131213+00:00 stdout F             Total: 8
2020-05-04T13:09:53.219131213+00:00 stdout F Resources:
2020-05-04T13:09:53.219131213+00:00 stdout F            Failed: 2
2020-05-04T13:09:53.219131213+00:00 stdout F           Changed: 6
2020-05-04T13:09:53.219131213+00:00 stdout F           Skipped: 72
2020-05-04T13:09:53.219131213+00:00 stdout F       Out of sync: 8
2020-05-04T13:09:53.219131213+00:00 stdout F             Total: 82
2020-05-04T13:09:53.219131213+00:00 stdout F Time:
2020-05-04T13:09:53.219131213+00:00 stdout F         File line: 0.00
2020-05-04T13:09:53.219131213+00:00 stdout F              File: 0.03
2020-05-04T13:09:53.219131213+00:00 stdout F     Pcmk property: 1.99
2020-05-04T13:09:53.219131213+00:00 stdout F          Last run: 1588597793
2020-05-04T13:09:53.219131213+00:00 stdout F    Config retrieval: 4.75
2020-05-04T13:09:53.219131213+00:00 stdout F             Total: 2.43
2020-05-04T13:09:53.219131213+00:00 stdout F Version:
2020-05-04T13:09:53.219131213+00:00 stdout F            Config: 1588597786
2020-05-04T13:09:53.219131213+00:00 stdout F            Puppet: 5.5.10
2020-05-04T13:09:53.527062554+00:00 stderr F + rc=1
2020-05-04T13:09:53.527062554+00:00 stderr F + set -e
2020-05-04T13:09:53.527062554+00:00 stderr F + set +ux

so until the containers are rebuilt with the same rpms as the base image we could hit this issue.

We're trying to solve this via https://bugs.launchpad.net/tripleo/+bug/1863442.

Emilien Macchi (emilienm) on 2020-07-28

Changed in tripleo:
milestone:	victoria-1 → victoria-3

Marios Andreou (marios-b) on 2020-11-03

Changed in tripleo:
milestone:	victoria-3 → wallaby-1

Marios Andreou (marios-b) on 2020-12-08

Changed in tripleo:
milestone:	wallaby-1 → wallaby-2

Marios Andreou (marios-b) on 2021-01-29

Changed in tripleo:
milestone:	wallaby-2 → wallaby-3

Marios Andreou (marios-b) on 2021-03-17

Changed in tripleo:
milestone:	wallaby-3 → wallaby-rc1

Marios Andreou (marios-b) on 2021-05-06

Changed in tripleo:
milestone:	wallaby-rc1 → xena-1

Revision history for this message

Marios Andreou (marios-b) wrote on 2021-05-07:

This is an automated action. Bug status has been set to 'Incomplete' and target milestone has been removed due to inactivity. If you disagree please re-set these values and reach out to us on freenode #tripleo

Changed in tripleo:
milestone:	xena-1 → none
status:	Triaged → Incomplete

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Changed in tripleo:
status:	New → Triaged
importance:	Undecided → High
assignee:	nobody → Bhagyashri Shewale (bhagyashri-shewale)
milestone:	none → victoria-1
assignee:	Bhagyashri Shewale (bhagyashri-shewale) → nobody
assignee:	nobody → amolkahat (amolkahat)