[2.8.11] juju fails to connect to lxd on host, and cannot start containers

[Alexander Balderson]

One machine is failing to launch lxd instances in a deployment, while all the other machines are having no problems.

Machine 3 in this deployment has the following error (taken from the machine-3.log):

2021-06-11 02:29:51 DEBUG juju.worker.dependency engine.go:598 "broker-tracker" manifold worker stopped: cannot create instance broker: creating LXD container manager: Get "http://unix.socket/1.0": EOF
2021-06-11 02:29:51 ERROR juju.worker.dependency engine.go:671 "broker-tracker" manifold worker returned unexpected error: cannot create instance broker: creating LXD container manager: Get "http://unix.socket/1.0": EOF
2021-06-11 02:29:51 DEBUG juju.worker.dependency engine.go:673 stack trace:
Get "http://unix.socket/1.0": EOF
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/container/lxd/connection.go:132:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/container/lxd/server.go:69:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/container/lxd/server.go:62:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/container/factory/factory.go:24: creating LXD container manager
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/container/broker/instance_broker.go:87:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/worker/containerbroker/broker.go:152: cannot create instance broker
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/worker/containerbroker/manifold.go:86:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/worker/fortress/occupy.go:63:
/home/jenkins/workspace/BuildJuju-amd64/_build/src/github.com/juju/juju/cmd/jujud/agent/engine/housing.go:93:
2021-06-11 02:29:55 DEBUG juju.container.lxd connection.go:167 using LXD snap socket: "/var/snap/lxd/common/lxd"
2021-06-11 02:29:55 DEBUG juju.worker.dependency engine.go:598 "broker-tracker" manifold worker stopped: cannot create instance broker: creating LXD container manager: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused
2021-06-11 02:29:55 ERROR juju.worker.dependency engine.go:671 "broker-tracker" manifold worker returned unexpected error: cannot create instance broker: creating LXD container manager: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused

It seems that juju is having isues connecting to lxd to create containers.
the logs from this particular run can be found at:
https://oil-jenkins.canonical.com/artifacts/d51933ca-990c-46a7-a9d4-285f31637680/generated/generated/kubernetes/juju-crashdump-kubernetes-2021-06-11-06.20.58.tar.gz
and all logs are available at:
https://oil-jenkins.canonical.com/artifacts/d51933ca-990c-46a7-a9d4-285f31637680/index.html

all testruns we have hit this this issue can be found:
https://solutions.qa.canonical.com/bugs/bugs/bug/1931726

[Ben Hoyt (benhoyt)]

Hi Alexander, are you still seeing this issue?

[Bas de Bruijne (basdbruijne)]

We are still seeing this occasionally.

You can find the occurrences here: https://solutions.qa.canonical.com/bugs/bugs/bug/1931726

Here are the logs for the latest hit: https://oil-jenkins.canonical.com/artifacts/52e3cd0a-5cca-4e66-9a21-22ba6b3e30bc/index.html

[John A Meinel (jameinel)]

I wonder if this is bug #1949705, which should be fixed in 2.8.13 which should be released today.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]