undercloud install failing for containers-multinode in the gate "OperationalError: (sqlite3.OperationalError) database is locked "

[wes hayutin]

The undercloud install is failing with a trace like

  2018-10-09 10:05:53.031 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather
  2018-10-09 10:05:53.032 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] than running sudo
  2018-10-09 10:05:53.032 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] fatal: [undercloud]: FAILED! => {"changed": true, "cmd": "sudo /usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.omzIHE-role-data --environment-file /tmp/ansible.0wn0Wj-prepare-param --cleanup partial 2> ../install-undercloud.log", "delta": "0:09:05.581756", "end": "2018-10-09 10:05:52.989193", "msg": "non-zero return code", "rc": 1, "start": "2018-10-09 09:56:47.407437", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
  2018-10-09 10:05:53.102 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:53.102 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] NO MORE HOSTS LEFT *************************************************************
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] PLAY RECAP *********************************************************************
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] undercloud : ok=201 changed=100 unreachable=0 failed=1
  2018-10-09 10:05:53.104 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ]
  2018-10-09 10:05:54.471 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning working directory /home/zuul/tripleo-heat-installer-templates
  2018-10-09 10:05:54.505 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Not cleaning ansible directory /home/zuul/undercloud-ansible-KT0LgE
  2018-10-09 10:05:54.525 19140 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Install artifact is located at /home/zuul/undercloud-install-20181009100553.tar.bzip2
  2018-10-09 10:05:54.526 19140 ERROR tripleoclient.v1.tripleo_deploy.Deploy [ ]
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  Deployment Failed!

  ERROR: Heat log files: /var/log/heat-launcher/undercloud_deploy-C7bRh1
...
  Traceback (most recent call last):
    File "/bin/ansible-playbook", line 118, in <module>
      exit_code = cli.run()
    File "/usr/lib/python2.7/site-packages/ansible/cli/playbook.py", line 122, in run
      results = pbex.run()
...
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
      reraise(type(exception), exception, tb=exc_tb, cause=cause)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
      context)
    File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
      cursor.execute(statement, parameters)
  OperationalError: (sqlite3.OperationalError) database is locked [SQL: "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"] (Background on this error at: http://sqlalche.me/e/e3q8)

  Error running playbook in directory: /tmp/tripleo-modify-image-playbook-BNg2Jf

many examples like

* http://logs.openstack.org/50/604750/3/gate/tripleo-ci-centos-7-containers-multinode/4405b2b/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-09_10_05_53_032

* http://logs.openstack.org/64/604664/1/gate/tripleo-ci-centos-7-containers-multinode/3f8c81e/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-08_20_46_55_914

* http://logs.openstack.org/98/604298/29/check/tripleo-ci-centos-7-containers-multinode/8804f81/logs/undercloud/home/zuul/install-undercloud.log.txt.gz#_2018-10-09_08_56_31_872

[wes hayutin (weshayutin)]

gate failure, so it's alert + promotion blocker

[wes hayutin (weshayutin)]

sqllite received a disconnect..

https://docs.sqlalchemy.org/en/latest/core/pooling.html#pool-disconnects

[wes hayutin (weshayutin)]

This looks like an ansible callback and ara issue

[wes hayutin (weshayutin)]

http://logs.openstack.org/89/608589/5/gate/tripleo-ci-centos-7-undercloud-containers/acf2c3e/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

[Marios Andreou (marios-b)]

o/ looking at current status [1] this is not currently happening (can't find any occurrences of it for today 09th and lots of success). Looking at the traces you linked in comment #4 and the description, i.e. [2] and [3] I can't see zuul reporting the failure for tripleo-ci-centos-7-containers-multinode on the gate jobs. Though clearly they failed as pointed to by those traces. So where did you get those from then please?.

For now going to remove the alert promotion-blocker until someone tells me I'm wrong and I'll add them back :) thanks

[1] http://cistatus.tripleo.org/gates/
[2] https://review.openstack.org/#/c/604664
[3] https://review.openstack.org/#/c/608589/

[Marios Andreou (marios-b)]

re added the tags after discussion with weshay just now. Another example of it happening in the gate today (added in the description). I updated the description and title too

[wes hayutin (weshayutin)]

http://logs.openstack.org/44/603944/3/gate/tripleo-ci-centos-7-containers-multinode/cabd937/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

http://logs.openstack.org/50/604750/3/gate/tripleo-ci-centos-7-containers-multinode/4405b2b/logs/undercloud/home/zuul/install-undercloud.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/609028

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/606986/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/609028