| 2021-06-04 21:18:47 |
james beedy |
description |
Hello,
We are experiencing a situation where juju models stop generating logs.
ubuntu@juju-controller-1:~$ sudo tail -f /var/log/juju/models/SSSLER@scania-penny-staging-e54c7d.log
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "42" (instance ID "e66hhy") to long poll group
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "43" (instance ID "fwsm4a") to long poll group
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 43 already started as instance "fwsm4a"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 36 already started as instance "kqqhng"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 39 already started as instance "qgmmtn"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 40 already started as instance "k33bxg"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 41 already started as instance "x3rp3w"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 42 already started as instance "e66hhy"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:475 provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
juju debug-log does show some interesting output
$ juju debug-log
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.dependency stack trace:
lease operation timed out
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:187: leadership failure
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:153:
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopping
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopped
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation preparing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation executing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation committing operation "resign leadership"
controller-0: 14:23:08 INFO juju.worker.provisioner Shutting down provisioner task machine-0
controller-0: 14:23:08 INFO juju.worker.logger logger worker stopped
controller-0: 14:23:08 INFO juju.worker.machineundertaker tearing down machine undertaker
As you can see both juju debug-log and the controller logs show the date of the last logs that were generated were yesterday.
Any insight on how to proceed here would be greatly appreciated.
Thank you! |
Hello,
We are experiencing a situation where juju models stop generating logs.
ubuntu@juju-controller-1:~$ sudo tail -f /var/log/juju/models/USER@staging-e54c7d.log
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "42" (instance ID "e66hhy") to long poll group
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "43" (instance ID "fwsm4a") to long poll group
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 43 already started as instance "fwsm4a"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 36 already started as instance "kqqhng"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 39 already started as instance "qgmmtn"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 40 already started as instance "k33bxg"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 41 already started as instance "x3rp3w"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 42 already started as instance "e66hhy"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:475 provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
juju debug-log does show some interesting output
$ juju debug-log
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.dependency stack trace:
lease operation timed out
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:187: leadership failure
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:153:
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopping
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopped
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation preparing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation executing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation committing operation "resign leadership"
controller-0: 14:23:08 INFO juju.worker.provisioner Shutting down provisioner task machine-0
controller-0: 14:23:08 INFO juju.worker.logger logger worker stopped
controller-0: 14:23:08 INFO juju.worker.machineundertaker tearing down machine undertaker
As you can see both juju debug-log and the controller logs show the date of the last logs that were generated were yesterday.
Any insight on how to proceed here would be greatly appreciated.
Thank you! |
|
