KvStore() is not mocked in unit_tests leading to random test failures

[Alex Kavanagh]

e.g. from: https://zuul.opendev.org/t/openstack/build/fa889c5f96cb4a76aaaed678ae1972eb

{4} unit_tests.test_neutron_ovs_context.TestDPDKUtils.test_resolve_dpdk_bridges [5.411990s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/charm-neutron-openvswitch/unit_tests/test_neutron_ovs_context.py", line 686, in test_resolve_dpdk_bridges
    self.assertEqual(context.resolve_dpdk_bridges(),

      File "hooks/neutron_ovs_context.py", line 381, in resolve_dpdk_bridges
    db = kv()

      File "hooks/charmhelpers/core/unitdata.py", line 524, in kv
    _KV = Storage()

      File "hooks/charmhelpers/core/unitdata.py", line 189, in __init__
    self._init()

      File "hooks/charmhelpers/core/unitdata.py", line 389, in _init
    )''')

    sqlite3.OperationalError: database is locked

[Aurelien Lourot (aurelien-lourot)]

Example of solutions in other charms:
https://review.opendev.org/c/openstack/charm-hacluster/+/769967
https://review.opendev.org/c/openstack/charm-percona-cluster/+/764384

[Alex Kavanagh (ajkavanagh)]

From vault charm unit test. KvStore needs mocking on it too:

{4} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_ext [5.153165s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/mock/mock.py", line 1346, in patched
    return func(*newargs, **newkeywargs)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/unit_tests/test_reactive_vault_handlers.py", line 739, in test_send_vault_url_and_ca_ext
    handlers.send_vault_url_and_ca()

      File "src/reactive/vault_handlers.py", line 615, in send_vault_url_and_ca
    vault_url = vault.get_api_url()

      File "src/lib/charm/vault.py", line 130, in get_vault_url
    if charms.reactive.is_state('vault.ssl.available'):

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 287, in is_state
    return is_flag_set(state)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 208, in is_flag_set
    return any_flags_set(flag)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 223, in any_flags_set
    active_flags = get_flags()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 232, in get_flags
    flags = unitdata.kv().getrange('reactive.states.', strip=True) or {}

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 524, in kv
    _KV = Storage()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 189, in __init__
    self._init()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 389, in _init
    )''')

    sqlite3.OperationalError: database is locked

Failing tests are:

{1} unit_tests.test_reactive_vault_handlers.TestHandlers.test_configure_vault [5.085344s] ... FAILED
{4} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_ext [5.153165s] ... FAILED
6} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_hostname [5.055302s] ... FAILED
{7} unit_tests.test_reactive_vault_handlers.TestHandlers.test_loadbalancer [5.200378s] ... FAILED

[Cory Johns (johnsca)]

You can make it use an in-memory database by setting this environment var:

UNIT_STATE_DB=:memory:

Or by doing the following before kv() is called:

charmhelpers.core.unitdata._KV = charmhelpers.core.unitdata.Storage(':memory:')

Perhaps this could be done by default in Zaza?

Reference: https://charm-helpers.readthedocs.io/en/latest/api/charmhelpers.core.unitdata.html#charmhelpers.core.unitdata.Storage

[Samuel Allan (samuelallan)]

I'm trying to reproduce the error for charm-vault locally, but cannot. Alex, do you have any more information about where you see this sqlite error, or if you can reliably reproduce it?

[Alex Kavanagh (ajkavanagh)]

Samuel, see the trace above in comment #2; it shows exactly where the lock occurred. However, it being a race-hazard bug means that it'll be hard to re-produce especially if you have a fast desktop/laptop as that'll make the locks smaller and thus less likely to occur at the same time.

One way to force a lock is just to do write a tight loop for say, 10 seconds, that constantly writes to the sql db (using the kv store) in at least two tests, limit the concurrency to 2 threads and that'll almost certainly result in a failure. However, it's probably not necessary to do that.

Either just use Cory's solution (#3) or better, make sure it is mocked out, as we are not actually testing the SQL database, and it's locking behaviour may change over versions. Thanks!

[Danny Cocks (dannycocks)]

After merging in https://github.com/juju/charm-helpers/pull/823 to charmhelpers, this shouldn't be a problem anymore.

[Alex Kavanagh (ajkavanagh)]

> After merging in https://github.com/juju/charm-helpers/pull/823 to charmhelpers, this shouldn't be a problem anymore.

This only affects the main development branch. As we use stable branches for stable versions, we may need to consider backporting the above change to stable branches.