Unit tests occasionally fail because SQLite hasn't been mocked

[Aurelien Lourot]

These tests occasionally fail on Zuul and it looks like SQLite should have been mocked:

{2} unit_tests.test_pcmk.TestPcmk.test_crm_update_resource_exists_in_kv_force_true [0.106502s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/charm-hacluster/.tox/py35/lib/python3.5/site-packages/mock/mock.py", line 1330, in patched
    return func(*args, **keywargs)

      File "/home/zuul/src/opendev.org/openstack/charm-hacluster/unit_tests/test_pcmk.py", line 267, in test_crm_update_resource_exists_in_kv_force_true
    db.set('res_test-IPaddr2', 'ef395293b1b7c29c5bf1c99774f75cf4')

      File "/home/zuul/src/opendev.org/openstack/charm-hacluster/charmhelpers/core/unitdata.py", line 294, in set
    (key, serialized))

    sqlite3.IntegrityError: UNIQUE constraint failed: kv.key

https://zuul.opendev.org/t/openstack/build/b62d2350a7a44466b8884475bcdadf1d
https://review.opendev.org/c/openstack/charm-hacluster/+/766575

[Aurelien Lourot (aurelien-lourot)]

https://review.opendev.org/c/openstack/charm-hacluster/+/769967

[Aurelien Lourot (aurelien-lourot)]

Similar issue seen on nova-cloud-controller:

https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/770415
https://zuul.opendev.org/t/openstack/build/d3ba0987f75c455285a645059f1ea8bf

    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/.tox/py36/lib/python3.6/site-packages/mock/mock.py", line 1346, in patched
    return func(*newargs, **newkeywargs)

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/unit_tests/test_nova_cc_hooks.py", line 465, in test_update_ssh_keys_and_notify_compute_units_ssh_migration
    hooks.update_ssh_keys_and_notify_compute_units()

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/hooks/nova_cc_hooks.py", line 740, in update_ssh_keys_and_notify_compute_units
    update_ssh_key(rid=rid, unit=unit)

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/hooks/nova_cc_hooks.py", line 853, in update_ssh_key
    resolved_hosts = ncc_utils.resolve_hosts_for(private_address, hostname)

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/hooks/nova_cc_utils.py", line 1337, in resolve_hosts_for
    db = unitdata.kv()

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/charmhelpers/core/unitdata.py", line 524, in kv
    _KV = Storage()

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/charmhelpers/core/unitdata.py", line 189, in __init__
    self._init()

      File "/home/zuul/src/opendev.org/openstack/charm-nova-cloud-controller/charmhelpers/core/unitdata.py", line 389, in _init
    )''')

    sqlite3.OperationalError: database is locked

[Cory Johns (johnsca)]

For future reference, you can make it use an in-memory database by setting this environment var:

UNIT_STATE_DB=:memory:

Or by doing the following before kv() is called:

charmhelpers.core.unitdata._KV = charmhelpers.core.unitdata.Storage(':memory:')

Perhaps this could be done by default in Zaza?

Reference: https://charm-helpers.readthedocs.io/en/latest/api/charmhelpers.core.unitdata.html#charmhelpers.core.unitdata.Storage

[Aurelien Lourot (aurelien-lourot)]

Still happening. This has just hit us in this review:
https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/679584
https://zuul.opendev.org/t/openstack/build/2af811c50c6a445a8e2e9e6855507600

[Alex Kavanagh (ajkavanagh)]

Review to fix on nova-cloud-controller: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/816510

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/816510
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/36ef217bc620bebe4475dbb1453d3c82ac5b9d73
Submitter: "Zuul (22348)"
Branch: master

commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863614

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863615

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863616

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863617

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863618

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863618
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/1d7cf2a04ead389f77009ad45ac321a42cf20886
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1d7cf2a04ead389f77009ad45ac321a42cf20886
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd
    (cherry picked from commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863614
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/3e3bd32d74ed77fea9b4e14c0d3ab5c15bb91c27
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 3e3bd32d74ed77fea9b4e14c0d3ab5c15bb91c27
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd
    (cherry picked from commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863615
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/7d544a69ae92a99b7a2a90fa1d07f83e1d8363f9
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7d544a69ae92a99b7a2a90fa1d07f83e1d8363f9
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd
    (cherry picked from commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863616
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/1358a613340bf05ad0c209d92e5f1ea942980402
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 1358a613340bf05ad0c209d92e5f1ea942980402
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd
    (cherry picked from commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/863617
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/eb14dba8fdaf9ce7533f48730bf3d3f8ce134c6a
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit eb14dba8fdaf9ce7533f48730bf3d3f8ce134c6a
Author: Alex Kavanagh <email address hidden>
Date: Wed Nov 3 13:58:00 2021 +0000

    Fix SQLite locking issue on unit tests

    stestr runs tests in parallel and this can cause issues with locking
    when SQLite is not mocked out properly and gets used in a test. This
    patch just switches Storage to use :memory: so that locking does not
    occur.

    Closes-Bug: #1908282
    Change-Id: Iaa1c7b78dee498e0cc6dc6fccf12e74f22225ecd
    (cherry picked from commit 36ef217bc620bebe4475dbb1453d3c82ac5b9d73)