We are experiencing a situation where juju models stop generating logs.
ubuntu@juju-controller-1:~$ sudo tail -f /<email address hidden>
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "42" (instance ID "e66hhy") to long poll group
2021-06-03 14:25:59 DEBUG juju.worker.instancepoller worker.go:534 moving machine "43" (instance ID "fwsm4a") to long poll group
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 43 already started as instance "fwsm4a"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 36 already started as instance "kqqhng"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 39 already started as instance "qgmmtn"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 40 already started as instance "k33bxg"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 41 already started as instance "x3rp3w"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:423 machine 42 already started as instance "e66hhy"
2021-06-03 14:26:00 INFO juju.worker.provisioner provisioner_task.go:475 provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
juju debug-log does show some interesting output
$ juju debug-log
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.dependency stack trace:
lease operation timed out
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:187: leadership failure
/var/lib/jenkins/workspace/BuildJuju-centos-amd64/_build/src/github.com/juju/juju/worker/leadership/tracker.go:153:
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopping
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopped
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation preparing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation executing operation "resign leadership"
unit-license-manager-agent-2: 14:22:59 WARNING juju.worker.uniter.operation we should run a leader-deposed hook here, but we can't yet
unit-license-manager-agent-2: 14:22:59 DEBUG juju.worker.uniter.operation committing operation "resign leadership"
controller-0: 14:23:08 INFO juju.worker.provisioner Shutting down provisioner task machine-0
controller-0: 14:23:08 INFO juju.worker.logger logger worker stopped
controller-0: 14:23:08 INFO juju.worker.machineundertaker tearing down machine undertaker
As you can see both juju debug-log and the controller logs show the date of the last logs that were generated were yesterday.
Any insight on how to proceed here would be greatly appreciated.
Hello,
We are experiencing a situation where juju models stop generating logs.
ubuntu@ juju-controller -1:~$ sudo tail -f /<email address hidden> instancepoller worker.go:534 moving machine "42" (instance ID "e66hhy") to long poll group instancepoller worker.go:534 moving machine "43" (instance ID "fwsm4a") to long poll group provisioner provisioner_ task.go: 423 machine 43 already started as instance "fwsm4a" provisioner provisioner_ task.go: 423 machine 36 already started as instance "kqqhng" provisioner provisioner_ task.go: 423 machine 39 already started as instance "qgmmtn" provisioner provisioner_ task.go: 423 machine 40 already started as instance "k33bxg" provisioner provisioner_ task.go: 423 machine 41 already started as instance "x3rp3w" provisioner provisioner_ task.go: 423 machine 42 already started as instance "e66hhy" provisioner provisioner_ task.go: 475 provisioner- harvest- mode is set to destroyed; unknown instances not stopped []
2021-06-03 14:25:59 DEBUG juju.worker.
2021-06-03 14:25:59 DEBUG juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
2021-06-03 14:26:00 INFO juju.worker.
juju debug-log does show some interesting output
$ juju debug-log manager- agent-2: 14:22:59 DEBUG juju.worker. dependency stack trace: jenkins/ workspace/ BuildJuju- centos- amd64/_ build/src/ github. com/juju/ juju/worker/ leadership/ tracker. go:187: leadership failure jenkins/ workspace/ BuildJuju- centos- amd64/_ build/src/ github. com/juju/ juju/worker/ leadership/ tracker. go:153: manager- agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopping manager- agent-2: 14:22:59 DEBUG juju.worker.uniter juju-run listener stopped manager- agent-2: 14:22:59 DEBUG juju.worker. uniter. operation preparing operation "resign leadership" manager- agent-2: 14:22:59 DEBUG juju.worker. uniter. operation executing operation "resign leadership" manager- agent-2: 14:22:59 WARNING juju.worker. uniter. operation we should run a leader-deposed hook here, but we can't yet manager- agent-2: 14:22:59 DEBUG juju.worker. uniter. operation committing operation "resign leadership" provisioner Shutting down provisioner task machine-0 machineundertak er tearing down machine undertaker
unit-license-
lease operation timed out
/var/lib/
/var/lib/
unit-license-
unit-license-
unit-license-
unit-license-
unit-license-
unit-license-
controller-0: 14:23:08 INFO juju.worker.
controller-0: 14:23:08 INFO juju.worker.logger logger worker stopped
controller-0: 14:23:08 INFO juju.worker.
As you can see both juju debug-log and the controller logs show the date of the last logs that were generated were yesterday.
Any insight on how to proceed here would be greatly appreciated.
Thank you!