upgrading from rev 39 to rev 51 failed.

[Seyeong Kim]

When upgrading rev 39 to rev 51, I faced issue like below [1]

test bundle is [2], after deploying them

juju upgrade-charm grafana

I can see the issue.

[1]
https://pastebin.ubuntu.com/p/WQsWj5TrwM/

[2]
- bundles

series: bionic
applications:
  elasticsearch:
    charm: cs:elasticsearch-51
    num_units: 1
    to:
    - "2"
  filebeat:
    charm: cs:filebeat-33
  grafana:
    charm: cs:grafana-39
    num_units: 1
    to:
    - "1"
  logrotated:
    charm: cs:~logrotate-charmers/logrotate-charm-5
  ntp:
    charm: cs:ntp-47
  prometheus2:
    charm: cs:prometheus2-25
    num_units: 1
    to:
    - "0"
machines:
  "0": {}
  "1": {}
  "2": {}
relations:
- - grafana:grafana-source
  - prometheus2:grafana-source
- - grafana:juju-info
  - ntp:juju-info
- - grafana:juju-info
  - filebeat:beats-host
- - logrotated:juju-info
  - grafana:juju-info
- - elasticsearch:juju-info
  - filebeat:beats-host
- - filebeat:elasticsearch
  - elasticsearch:client

[Xav Paice (xavpaice)]

Reproduced on Bionic. Xenial works fine.

Appears to be an issue starting Grafana:
2021-10-19 21:23:54 INFO unit.grafana-b/0.juju-log server.go:327 Reactive main running for hook upgrade-charm
2021-10-19 21:23:55 ERROR unit.grafana-b/0.juju-log server.go:327 Unable to find implementation for relation: requires of register-application
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Initializing Snap Layer
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/snap.py:117:upgrade_charm
2021-10-19 21:23:56 WARNING unit.grafana-b/0.upgrade-charm logger.go:60 All snaps up to date.
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/tls_client.py:186:remove_states
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:652:wipe_nrpe_checks
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:1186:check_adminuser
2021-10-19 21:23:56 INFO unit.grafana-b/0.juju-log server.go:327 Checking for existing users, loop 1 of 6
2021-10-19 21:24:01 INFO unit.grafana-b/0.juju-log server.go:327 check_adminuser::sqlite3.OperationError: no such table: user
2021-10-19 21:24:01 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
2021-10-19 21:24:01 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
2021-10-19 21:24:02 INFO juju.worker.uniter.operation runhook.go:152 ran "upgrade-charm" hook (via explicit, bespoke hook script)
2021-10-19 21:24:02 INFO juju.worker.uniter resolver.go:154 found queued "config-changed" hook
2021-10-19 21:24:03 INFO unit.grafana-b/0.juju-log server.go:327 Reactive main running for hook config-changed
2021-10-19 21:24:03 ERROR unit.grafana-b/0.juju-log server.go:327 Unable to find implementation for relation: requires of register-application
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Initializing Snap Layer
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:323:config_changed
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:483:setup_grafana
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:652:wipe_nrpe_checks
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Invoking reactive handler: reactive/grafana.py:585:restart_grafana
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Starting grafana-server
2021-10-19 21:24:04 INFO unit.grafana-b/0.juju-log server.go:327 Retrying '_block_until_port_open' 10 more times (delay=2)
2021-10-...

[Xav Paice (xavpaice)]

Workaround: `chown grafana.grafana /var/lib/grafana/grafana.db && systemctl restart grafana-server`

This appears to be a permission issue on the sqlite db:

ls -l /var/lib/grafana/grafana.db
-rw-r--r-- 1 root root 0 Oct 19 21:20 /var/lib/grafana/grafana.db

[Xav Paice (xavpaice)]

Noted in (bionic) cs:grafana-39, after the install is completed, the grafana.db is root owned and Grafana was not started. Even after making a config-changed hook (enabled debug) the service was still stopped, although hooks were marked as OK.

[James Troup (elmo)]

To clarify, the fixed charm is available from the `edge` channel on charmhub. It'll be released to stable along with the rest of the legacy LMA stack in April.