In this other problematic run [0] however we don't see all these glitches. Instead the first failure seems to be:
Jul 16 12:10:27 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Started LXD - main daemon.
Jul 16 12:10:41 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Jul 16 12:10:57 juju-d19098-zaza-f68227589c9c-21 snapd[3336]: stateengine.go:150: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/names?confinement=strict%2Cclassic: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Jul 16 12:11:32 juju-d19098-zaza-f68227589c9c-21 snapd[3336]: daemon.go:542: gracefully waiting for running hooks
Jul 16 12:11:32 juju-d19098-zaza-f68227589c9c-21 snapd[3336]: daemon.go:544: done waiting for running hooks
Jul 16 12:11:32 juju-d19098-zaza-f68227589c9c-21 snapd[3336]: daemon stop requested to wait for socket activation
Jul 16 12:11:56 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Jul 16 12:11:56 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Synchronized to time server 91.189.91.157:123 (ntp.ubuntu.com).
Jul 16 12:13:25 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Starting Snap Daemon...
Jul 16 12:13:25 juju-d19098-zaza-f68227589c9c-21 snapd[10081]: AppArmor status: apparmor is enabled and all features are available
Jul 16 12:13:25 juju-d19098-zaza-f68227589c9c-21 snapd[10081]: daemon.go:343: started snapd/2.45.1+18.04 (series 16; classic) ubuntu/18.04 (amd64) linux/4.15.0-111-generic.
Jul 16 12:13:25 juju-d19098-zaza-f68227589c9c-21 snapd[10081]: daemon.go:436: adjusting startup timeout by 30s (pessimistic estimate of 30s plus 5s per snap)
Jul 16 12:13:25 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Started Snap Daemon.
Jul 16 12:13:43 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Jul 16 12:14:26 juju-d19098-zaza-f68227589c9c-21 snapd[10081]: api.go:990: Installing snap "core" revision unset
Jul 16 12:14:58 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Jul 16 12:15:08 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.89.198:123 (ntp.ubuntu.com).
Jul 16 12:15:18 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com).
Jul 16 12:15:18 juju-d19098-zaza-f68227589c9c-21 snapd[10081]: 2020/07/16 12:15:18 Unsolicited response received on idle HTTP channel starting with "HTTP/1.0 408 Request Time-out\r\nCache-Control: no-cache\r\nConnection: close\r\nContent-Type: text/html\r\n\r\n408 Request Time-out
\nYour browser didn't send a complete request in time.\n\n"; err=
Jul 16 12:15:28 juju-d19098-zaza-f68227589c9c-21 systemd-timesyncd[526]: Timed out waiting for reply from 91.189.94.4:123 (ntp.ubuntu.com).
Jul 16 12:16:32 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Reloading.
Jul 16 12:16:32 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Mounting Mount unit for core, revision 9436...
Jul 16 12:16:32 juju-d19098-zaza-f68227589c9c-21 systemd[1]: Mounted Mount unit for core, revision 9436.
So I think the issue is sometimes a network hiccup on "snap install" and then we could just retry, but sometimes it's a hiccup way earlier leading to some missing keys preventing snap from doing some mounts. In the latter case I'm not sure yet how to recover.
0: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-trilio-horizon-plugin/740934/5/6341/index.html