upgrade-series prepare jammy failed at 'sqlite3 database is locked '

[Chuan Li]

charm: ch:ceph-mon, channel: quincy/stable, revision: 149
series: focal

When upgrades series from Focal to Jammy, 'juju upgrade-series prepare jammy' hangs at 'pre-series-upgrade hook running'

juju-unit-log:

2023-02-04 11:17:54 ERROR unit.ceph-mon/0.juju-log server.go:327 Uncaught exception while in charm code:
Traceback (most recent call last):
  File "./src/charm.py", line 282, in <module>
    main(CephMonCharm)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/main.py", line 435, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/main.py", line 144, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 355, in emit
    framework._emit(event) # noqa
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 824, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/ops/framework.py", line 899, in _reemit
    custom_handler(event)
  File "./src/charm.py", line 85, in on_pre_series_upgrade
    hooks.pre_series_upgrade()
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_hooks.py", line 1218, in pre_series_upgrade
    set_unit_paused()
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/charmhelpers/contrib/openstack/utils.py", line 1560, in set_unit_paused
    with unitdata.HookData()() as t:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/charmhelpers/core/unitdata.py", line 473, in __call__
    with self.kv.hook_scope(hook_name):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/var/lib/juju/agents/unit-ceph-mon-0/charm/venv/charmhelpers/core/unitdata.py", line 361, in hook_scope
    self.cursor.execute(
sqlite3.OperationalError: database is locked

Brief steps:

1) deploy ceph-mon(cs:ceph-mon-51) onto a focal.
ceph is Octopus

2) juju refresh --switch ch:ceph-mon --channel quincy/stable ceph-mon

3) upgrade from Octopus to Pacific
juju config ceph-mon source=cloud:focal-wallaby

4)
juju set-series ceph-mon jammy
juju upgrade-series 0 prepare jammy

Detail steps: https://pastebin.canonical.com/p/R5MkdJnGC8/

[Diko Parvanov (dparv)]

Also affecting a quincy ceph-mon on focal with upgrade to jammy, without skipping releases.

[Diko Parvanov (dparv)]

This seems to be related to the juju operator framework. Adding a time.sleep(100) before the set_unit_paused() function that fails -

-rw------- 1 root root 40960 Feb 23 15:38 .unit-state.db
-rw------- 1 root root 16928 Feb 23 15:50 .unit-state.db-journal

$ sqlite3
SQLite version 3.31.1 2020-01-27 19:55:54
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open .unit-state.db
sqlite> create table if not exists hooks ( version integer primary key autoincrement, hook text, date text ) ;
sqlite> ^Ceate table if not exists hooks ( version integer primary key autoincrement, hook text, date text ) ;
sqlite> drop table hooks;
Error: database is locked

lsof | grep .unit-state.db
charm.py 3965098 root 3uw REG 253,1 40960 17709992 /var/lib/juju/agents/unit-ceph-mon-0/charm/.unit-state.db
charm.py 3965098 root 4u REG 253,1 16928 17697644 /var/lib/juju/agents/unit-ceph-mon-0/charm/.unit-state.db-journal

seems that juju is locking the db on any hook/event execution preventing the charmhelpers library to amend the state db

this is happening on juju 2.9.38

[Diko Parvanov (dparv)]

Possibly a better place for this to be tracked: https://github.com/canonical/operator/issues/909

[Diko Parvanov (dparv)]

Workaround to unblock the upgrade:

juju run -a ceph-mon "sudo sed -i 's/.unit-state.db/.unit-state-ch.db/g' /var/lib/juju/agents/unit-ceph-mon-*/charm/venv/charmhelpers/core/unitdata.py"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/876901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/876901
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/616c4e3367f2816774d8629396340a5b3b48f680
Submitter: "Zuul (22348)"
Branch: master

commit 616c4e3367f2816774d8629396340a5b3b48f680
Author: Luciano Lo Giudice <email address hidden>
Date: Wed Mar 8 19:31:35 2023 -0300

    Use a different name for the local key/value store

    The operator framework and charmhelpers use the same path for the
    local K/V store, which causes problems when running certain hooks
    like 'pre-series-upgrade'. In order to work around this issue, this
    patchset makes the charmhelpers lib use a different path, while
    migrating the DB file before doing so.

    Closes-Bug: #2005137
    Change-Id: Ic2e024371ff431888731753d29fff8538232009a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/quincy.2
Review: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ceph-mon/+/877397
Committed: https://opendev.org/openstack/charm-ceph-mon/commit/ca074377bbbad4c1ebd800c7b4fe8f8b9700a8a9
Submitter: "Zuul (22348)"
Branch: stable/quincy.2

commit ca074377bbbad4c1ebd800c7b4fe8f8b9700a8a9
Author: Luciano Lo Giudice <email address hidden>
Date: Wed Mar 8 19:31:35 2023 -0300

    Use a different name for the local key/value store

    The operator framework and charmhelpers use the same path for the
    local K/V store, which causes problems when running certain hooks
    like 'pre-series-upgrade'. In order to work around this issue, this
    patchset makes the charmhelpers lib use a different path, while
    migrating the DB file before doing so.

    Closes-Bug: #2005137
    Change-Id: Ic2e024371ff431888731753d29fff8538232009a
    (cherry picked from commit 616c4e3367f2816774d8629396340a5b3b48f680)

[Felipe Reyes (freyes)]

Charm-helpers PR - https://github.com/juju/charm-helpers/pull/759