undercloud install failing for containers-multinode in the gate "OperationalError: (sqlite3.OperationalError) database is locked "

Bug #1796764 reported by wes hayutin on 2018-10-08
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

The undercloud install is failing with a trace like

  2018-10-09 10:05:53.031 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather
  2018-10-09 10:05:53.032 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] than running sudo
  2018-10-09 10:05:53.032 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.omzIHE-role-data --environment-file /tmp/ansible.0wn0Wj-prepare-param --cleanup partial 2> ../install-undercloud.log", "delta": "0:09:05.581756", "end": "2018-10-09 10:05:52.989193", "msg": "non-zero return code", "rc": 1, "start": "2018-10-09 09:56:47.407437", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
  2018-10-09 10:05:53.102 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:53.102 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] NO MORE HOSTS LEFT *************************************************************
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] PLAY RECAP *********************************************************************
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] undercloud : ok=201 changed=100 unreachable=0 failed=1
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:54.471 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
  2018-10-09 10:05:54.505 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/zuul/undercloud-ansible-KT0LgE
  2018-10-09 10:05:54.525 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/zuul/undercloud-install-20181009100553.tar.bzip2
  2018-10-09 10:05:54.526 19140 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ]
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  Deployment Failed!

  ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-C7bRh1
...
  Traceback (most recent call last):
    File "/bin/ansible-playbook", line 118, in <module>
      exit_code = cli.run()
    File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 122, in run
      results = pbex.run()
...
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
      context)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
      cursor.execute(statement, parameters)
  OperationalError: (sqlite3.OperationalError) database is locked [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/e3q8)

  Error running playbook in directory: /tmp/tripleo-modify-image-playbook-BNg2Jf

many examples like

* http://logs.openstack.org/50/604750/3/gate/tripleo-ci-centos-7-containers-multinode/4405b2b/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-09_10_05_53_032

* http://logs.openstack.org/64/604664/1/gate/tripleo-ci-centos-7-containers-multinode/3f8c81e/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-08_20_46_55_914

* http://logs.openstack.org/98/604298/29/check/tripleo-ci-centos-7-containers-multinode/8804f81/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-09_08_56_31_872

wes hayutin (weshayutin) wrote :

gate failure, so it's alert + promotion blocker

wes hayutin (weshayutin) wrote :

This looks like an ansible callback and ara issue

Marios Andreou (marios-b) wrote :

o/ looking at current status [1] this is not currently happening (can't find any occurrences of it for today 09th and lots of success). Looking at the traces you linked in comment #4 and the description, i.e. [2] and [3] I can't see zuul reporting the failure for tripleo-ci-centos-7-containers-multinode on the gate jobs. Though clearly they failed as pointed to by those traces. So where did you get those from then please?.

For now going to remove the alert promotion-blocker until someone tells me I'm wrong and I'll add them back :) thanks

[1] http://cistatus.tripleo.org/gates/
[2] https://review.openstack.org/#/c/604664
[3] https://review.openstack.org/#/c/608589/

tags: removed: alert promotion-blocker
tags: added: alert ci promotion-blocker
description: updated
Marios Andreou (marios-b) wrote :

re added the tags after discussion with weshay just now. Another example of it happening in the gate today (added in the description). I updated the description and title too

summary: - OperationalError: (sqlite3.OperationalError) database is locked [SQL:
- "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"]
- (Background on this error at: http://sqlalche.me/e/e3q8)
+ undercloud install failing for containers-multinode in the gate
+ "OperationalError: (sqlite3.OperationalError) database is locked "
description: updated
Alex Schultz (alex-schultz) wrote :
Changed in tripleo:
status: Triaged → Fix Released

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/609028

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers