graylog fails installation in offlline even with snap resources attached

[Yoshi Kadokawa]

When in an offline environment, Graylog charm will fail during the installation phase during the snap installation.
Both core and graylog snaps are specified with the resource attached to the charm.
And confirmed that the snaps are indeed installed.

$ juju run --unit graylog/0 -- sudo snap list
Name Version Rev Tracking Publisher Notes
core 16-2.50.1 x6 - - core
core18 20210507 2066 latest/stable canonical* base
graylog 3.0.1 x1 - - -
lxd 4.0.6 20326 4.0/stable/… canonical* -
snapd 2.50.1 12057 latest/stable canonical* snapd

However, the charm is in error state with the following error message in the logs.

2021-07-11 12:59:07 INFO juju-log Reactive main running for hook install
2021-07-11 12:59:07 INFO juju-log Initializing Leadership Layer (is leader)
2021-07-11 12:59:08 INFO juju-log Initializing Snap Layer
2021-07-11 12:59:10 INFO juju-log Installing /var/lib/juju/agents/unit-graylog-0/resources/core/core.snap from local resource
2021-07-11 12:59:15 INFO juju-log Invoking reactive handler: reactive/graylog.py:101:install_graylog
2021-07-11 12:59:18 INFO juju-log Installing /var/lib/juju/agents/unit-graylog-0/resources/graylog/graylog.snap from local resource
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: reactive/graylog.py:247:get_leader_passwords
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: reactive/graylog.py:258:appoint_master
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: reactive/snap.py:329:change_snapd_refresh
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: hooks/relations/http/provides.py:15:broken:website
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: hooks/relations/tls-certificates/requires.py:109:broken:certificates
2021-07-11 12:59:27 INFO juju-log Invoking reactive handler: reactive/graylog.py:115:refresh_graylog
2021-07-11 12:59:27 INFO juju-log Get channel for snap graylog
2021-07-11 12:59:41 INFO juju.worker.diskmanager diskmanager.go:67 block devices changed: []storage.BlockDevice{storage.BlockDevice{DeviceName:"loop0", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:
"", Size:0x37, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/core18/2066", SerialId:""}, storage.BlockDevice{DeviceName:"loop1", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x4
3, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/lxd/20326", SerialId:""}, storage.BlockDevice{DeviceName:"loop2", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x20, FilesystemT
ype:"squashfs", InUse:true, MountPoint:"/snap/snapd/12057", SerialId:""}, storage.BlockDevice{DeviceName:"loop3", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x63, FilesystemType:"squashf
s", InUse:true, MountPoint:"/snap/core/x5", SerialId:""}, storage.BlockDevice{DeviceName:"loop4", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x96, FilesystemType:"squashfs", InUse:true,
MountPoint:"/snap/graylog/17", SerialId:""}, storage.BlockDevice{DeviceName:"loop6", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x63, FilesystemType:"squashfs", InUse:true, MountPoint:"/
snap/core/x6", SerialId:""}, storage.BlockDevice{DeviceName:"loop7", DeviceLinks:[]string(nil), Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x96, FilesystemType:"squashfs", InUse:true, MountPoint:"/snap/graylog/x1"
, SerialId:""}, storage.BlockDevice{DeviceName:"vda", DeviceLinks:[]string{"/dev/disk/by-dname/vda", "/dev/disk/by-id/virtio-61dc1b4c-52c8-4154-8", "/dev/disk/by-path/pci-0000:00:07.0", "/dev/disk/by-path/virtio-pci-0000:00:07.0"},
Label:"", UUID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x4a81, FilesystemType:"", InUse:true, MountPoint:"", SerialId:"61dc1b4c-52c8-4154-8"}, storage.BlockDevice{DeviceName:"vda1", DeviceLinks:[]string{"/dev/disk/by-dname/vd
a-part1", "/dev/disk/by-id/virtio-61dc1b4c-52c8-4154-8-part1", "/dev/disk/by-partuuid/0c6c0757-5a14-4f5f-955a-f2a735cb7825", "/dev/disk/by-path/pci-0000:00:07.0-part1", "/dev/disk/by-path/virtio-pci-0000:00:07.0-part1"}, Label:"", U
UID:"", HardwareId:"", WWN:"", BusAddress:"", Size:0x1, FilesystemType:"", InUse:false, MountPoint:"", SerialId:"61dc1b4c-52c8-4154-8"}, storage.BlockDevice{DeviceName:"vda2", DeviceLinks:[]string{"/dev/disk/by-dname/vda-part2", "/d
ev/disk/by-id/virtio-61dc1b4c-52c8-4154-8-part2", "/dev/disk/by-label/root", "/dev/disk/by-partuuid/095902ab-00da-4d96-9a80-ca6eab2b81b8", "/dev/disk/by-path/pci-0000:00:07.0-part2", "/dev/disk/by-path/virtio-pci-0000:00:07.0-part2"
, "/dev/disk/by-uuid/5c108b1c-949e-4772-9609-abbeca270a2e"}, Label:"root", UUID:"5c108b1c-949e-4772-9609-abbeca270a2e", HardwareId:"", WWN:"", BusAddress:"", Size:0x4a7f, FilesystemType:"ext4", InUse:true, MountPoint:"/", SerialId:"
61dc1b4c-52c8-4154-8"}}
2021-07-11 13:00:07 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-graylog-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-graylog-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-graylog-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-graylog-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-graylog-0/charm/reactive/graylog.py", line 125, in refresh_graylog
    if get_channel_change_status() == ChannelChangeStatus.MINOR_CHANGE:
  File "lib/charms/layer/graylog/snap_change.py", line 46, in get_channel_change_status
    cur_channel = snap.get_installed_channel(SNAP_NAME)
  File "lib/charms/layer/snap.py", line 303, in get_installed_channel
    return subprocess.check_output(cmd).decode("utf-8", errors="replace").partition("tracking:")[-1].split()[0]
IndexError: list index out of range

It fails to retrieve a "tracking" info from "snap info graylog", which is not available from snaps installed locally.

$ snap info graylog
name: graylog
summary: Graylog server snap, https://www.graylog.org/
publisher: –
license: unset
description: |
  Graylog is a full-featured log management system allowing for centralized
  management, processing and alerting on logs from applications and systems.

  At a miniumum update the configuration at /var/snap/graylog/common/server.conf
  to set the admin password and mongodb connection string. Connection
  information for elastic search is also needed, refer to the Graylog
  documentation.
services:
  graylog: simple, enabled, active
refresh-date: today at 12:59 UTC
installed: 3.0.1 (x1) 157MB -

[Yoshi Kadokawa (yoshikadokawa)]

I believe the issue is in ./reactive/graylog.py with refresh_graylog() and report_status() functions, where it tries to get snap's channel information.
I think this should not be called at all when the graylog snap is installed via a local resource.

As of now I couldn't find a smart way to workaround this issue.

[Yoshi Kadokawa (yoshikadokawa)]

Since I could not find a workaround for this and blocking a deployment, I'm subscribing to field-high.

[Yoshi Kadokawa (yoshikadokawa)]

Sorry, field-high was wrong. Since there is no workaround at the moment, subscribing to field-critical

[Yoshi Kadokawa (yoshikadokawa)]

also related with this bug.
https://bugs.launchpad.net/layer-snap/+bug/1938271

[Diko Parvanov (dparv)]

Once snap layer bug 1938271 gets fixed with https://code.launchpad.net/~yoshikadokawa/layer-snap/+git/layer-snap/+merge/406295 and get_installed_channel() is returning None when a local snap is installed, we need to fix graylog charm with something like this:

https://code.launchpad.net/~dparv/charm-graylog/+git/charm-graylog/+merge/407072

I have tested this solution and it works.

[Nobuto Murata (nobuto)]

> Once snap layer bug 1938271 gets fixed with https://code.launchpad.net/~yoshikadokawa/layer-snap/+git/layer-snap/+merge/406295 and get_installed_channel() is returning None when a local snap is installed, we need to fix graylog charm with something like this:
>
> https://code.launchpad.net/~dparv/charm-graylog/+git/charm-graylog/+merge/407072
>
> I have tested this solution and it works.

Both of the merge proposals have been merged so marking this as Fix Committed.