glance-simplestreams-sync stuck waiting for lock

[Jason Hobbs]

Started sync at 4:15, we timed it out at 8:15.

DEBUG * 09-04 04:15:05 [PID:27189] * root * command: /snap/bin/simplestreams.sstream-mirror-glance -vv --keep --max 1 --content-id auto.sync --cloud-name glance-simplestreams-sync-openstack --path streams/v1/index.sjson --name-prefix auto-sync/ --keyring /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg --log-file /var/snap/simplestreams/common/sstream-mirror-glance.log --output-swift simplestreams/data/ http://cloud-images.ubuntu.com/releases/ release~(trusty|xenial|bionic|focal) arch~(x86_64|amd64) ftype~(disk1.img|disk.img)
INFO * 09-04 04:16:02 [PID:27311] * root * glance-simplestreams-sync started.
INFO * 09-04 04:16:02 [PID:27311] * root * /var/run/glance-simplestreams-sync.pid is locked, exiting

/var/log/glance-simplestreams-sync.log has the pid is locked error over and over, that's it.

https://solutions.qa.canonical.com/openstack/testRun/5e46e184-4231-496b-b161-e1b74e724032

[Marian Gasparovic (marosg)]

another run with nothing in log although sync took over 4 hours
https://solutions.qa.canonical.com/testruns/testRun/a18a1ea1-583f-4ffa-8bdf-5e83938d012a

[Michael Skalka (mskalka)]

Given that we are seeing this approximately once every other day [0] I am escalating this to field-high.

0. https://solutions.qa.canonical.com/bugs/bugs/bug/1894239

[James Page (james-page)]

can we collect the actual simplestreams log file from the deployments:

/var/snap/simplestreams/common/sstream-mirror-glance.log

that will give us a clue as to what is taking so long.

[Michael Skalka (mskalka)]

@james we don't have the contents of /var/snap int he existing crashdump but we do have /var/log/glance-simplestreams-sync.log which can be found in the crashdump attached to the run: https://oil-jenkins.canonical.com/artifacts/ae7164f0-08fd-4cd8-a4a9-0c7ef9021457/generated/generated/openstack/juju-crashdump-openstack-2020-10-21-16.24.22.tar.gz

If there is additional data in the snap log that into our log capture.

[James Page (james-page)]

logging to /var/snap/** is sensible as simplestreams is provided as a strictly confined snap.

This won't be the only tool that we ship this way to maybe we need to start collecting log files from under /var/snap as part of juju-crashdump?

[James Page (james-page)]

"/var/log/glance-simplestreams-sync.log" us the charm managed part of the sync process - managing concurrency etc.

[Launchpad Janitor (janitor)]

[Expired for OpenStack glance-simplestreams-sync charm because there has been no activity for 60 days.]

[Bas de Bruijne (basdbruijne)]

We are seeing this again here: https://solutions.qa.canonical.com/testruns/testRun/356ab384-29ad-4cea-9d0a-1d74bdcd90a1

`/var/snap/` is in the crashdump now but doesn't give further indication of what is failing.

Crashdump can be found here:
https://oil-jenkins.canonical.com/artifacts/356ab384-29ad-4cea-9d0a-1d74bdcd90a1/index.html

[Jadon Naas (jadonn)]

The last occurrence of this bug was in December 2022 it seems. Has this problem shown up anywhere else since then?