localhost (lxd) controller fails to restart after reboot

[Martin Hilton]

I have a localhost (lxd) controller which refuses to restart following a reboot of the host machine.

From the machine-0.log it seems there is a problem starting up the database:

2021-02-10 09:09:41 INFO juju.cmd supercommand.go:54 running jujud [2.8.8 0 1d2677ae1d65f10bd15e9e29ab5e0f0cca70898a gc go1.14.10]
2021-02-10 09:09:41 DEBUG juju.cmd supercommand.go:55 args: []string{"/var/lib/juju/tools/machine-0/jujud", "machine", "--data-dir", "/var/lib/juju", "--machine-id"
, "0", "--debug"}
2021-02-10 09:09:41 DEBUG juju.utils gomaxprocs.go:24 setting GOMAXPROCS to 16
2021-02-10 09:09:41 DEBUG juju.agent agent.go:582 read agent config, format "2.0"
2021-02-10 09:09:41 INFO juju.cmd.jujud agent.go:139 setting logging config to "<root>=INFO"
2021-02-10 09:09:41 INFO juju.worker.upgradesteps worker.go:70 upgrade steps for 2.8.8 have already been run.
2021-02-10 09:09:41 ERROR juju.mongo mongo.go:649 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/
transparent_hugepage/enabled: permission denied
2021-02-10 09:09:41 ERROR juju.mongo mongo.go:649 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/t
ransparent_hugepage/defrag: permission denied
2021-02-10 09:09:41 ERROR juju.mongo mongo.go:649 could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does
 not exist, will not set "16384"
2021-02-10 09:09:41 ERROR juju.mongo mongo.go:649 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_
max_backlog" does not exist, will not set "1000"
2021-02-10 09:09:41 INFO juju.mongo mongo.go:484 Ensuring mongo server is running; data directory /var/lib/juju; port 37017
2021-02-10 09:09:41 WARNING juju.mongo service.go:326 configuring mongod with --noauth flag enabled
2021-02-10 09:09:41 INFO juju.packaging manager.go:103 installing "mongodb-server-core" via "apt"
2021-02-10 09:09:41 INFO juju.packaging.manager utils.go:64 Running: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --ass
ume-yes --quiet install mongodb-server-core
2021-02-10 09:09:41 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [5e7c03] "machine-0" cannot open api: unable to
 connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2021-02-10 09:09:42 INFO juju.packaging manager.go:103 installing "mongodb-clients" via "apt"
2021-02-10 09:09:42 INFO juju.packaging.manager utils.go:64 Running: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install mongodb-clients
2021-02-10 09:09:43 ERROR juju.service.systemd service.go:127 failed to deserialize conf for application "juju-db": missing Desc
2021-02-10 09:09:43 ERROR juju.worker.dependency engine.go:671 "state" manifold worker returned unexpected error: failed to deserialize conf for application "juju-db": missing Desc
2021-02-10 09:09:45 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [5e7c03] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused
2021-02-10 09:09:47 ERROR juju.mongo mongo.go:649 could not set the value of "/sys/kernel/mm/transparent_hugepage/enabled" to "never" because of: open /sys/kernel/mm/transparent_hugepage/enabled: permission denied
2021-02-10 09:09:47 ERROR juju.mongo mongo.go:649 could not set the value of "/sys/kernel/mm/transparent_hugepage/defrag" to "never" because of: open /sys/kernel/mm/transparent_hugepage/defrag: permission denied
2021-02-10 09:09:47 ERROR juju.mongo mongo.go:649 could not set the value of "/proc/sys/net/core/somaxconn" to "16384" because of: "/proc/sys/net/core/somaxconn" does not exist, will not set "16384"
2021-02-10 09:09:47 ERROR juju.mongo mongo.go:649 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"
2021-02-10 09:09:47 INFO juju.mongo mongo.go:484 Ensuring mongo server is running; data directory /var/lib/juju; port 37017
2021-02-10 09:09:47 WARNING juju.mongo service.go:326 configuring mongod with --noauth flag enabled
2021-02-10 09:09:47 INFO juju.packaging manager.go:103 installing "mongodb-server-core" via "apt"
2021-02-10 09:09:47 INFO juju.packaging.manager utils.go:64 Running: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install mongodb-server-core
2021-02-10 09:09:47 INFO juju.packaging manager.go:103 installing "mongodb-clients" via "apt"
2021-02-10 09:09:47 INFO juju.packaging.manager utils.go:64 Running: apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::Options::=--force-unsafe-io --assume-yes --quiet install mongodb-clients
2021-02-10 09:09:48 ERROR juju.service.systemd service.go:127 failed to deserialize conf for application "juju-db": missing Desc
2021-02-10 09:09:48 ERROR juju.worker.dependency engine.go:671 "state" manifold worker returned unexpected error: failed to deserialize conf for application "juju-db": missing Desc
2021-02-10 09:09:50 ERROR juju.worker.dependency engine.go:671 "api-caller" manifold worker returned unexpected error: [5e7c03] "machine-0" cannot open api: unable to connect to API: dial tcp 127.0.0.1:17070: connect: connection refused

Indeed the /etc/systemd/system/juju-db.service file is completely empty, which I assume explains why that won't start.

-rw-r--r-- 1 root root 0 Feb 10 09:08 /etc/systemd/system/juju-db.service

[John A Meinel (jameinel)]

2021-02-10 09:09:47 ERROR juju.mongo mongo.go:649 could not set the value of "/proc/sys/net/core/netdev_max_backlog" to "1000" because of: "/proc/sys/net/core/netdev_max_backlog" does not exist, will not set "1000"

is not a problem. But:
2021-02-10 09:09:48 ERROR juju.service.systemd service.go:127 failed to deserialize conf for application "juju-db": missing Desc
2021-02-10 09:09:48 ERROR juju.worker.dependency engine.go:671 "state" manifold worker returned unexpected error: failed to deserialize conf for application "juju-db": missing Desc

seem problematic. And as you mention, it is empty.

I know we have code that says "make sure mongo is up and running" we could update the check so it doesn't just require the service file to exist, but also that it is non-empty.