Juju controller agent-version: 2.1.2 becomes unresponsive (e.g. juju status times out).
This happened just after an apt upgrade installed juju-mongo-tools3.2
Upon inspection of the controller logs show:
Jan 17 22:37:21 juju-3bc6d2-controller-0 systemd[1]: Started juju state database.
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod[1519]: 2018-01-17T22:37:21.860+0000 W CONTROL [main] No SSL certificate validation can be performed since no CA file has been provided; please specify an sslCAFile parameter
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] MongoDB starting : pid=1519 port=37017 dbpath=/var/lib/juju/db 64-bit host=juju-3bc6d2-controller-0
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] db version v3.2.15
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] git version: e11e3c1b9c9ce3f7b4a79493e16f5e4504e01140
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] allocator: tcmalloc
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] modules: none
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] build environment:
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] distarch: x86_64
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] target_arch: x86_64
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] options: { net: { ipv6: true, port: 37017, ssl: { PEMKeyFile: "/var/lib/juju/server.pem", PEMKeyPassword: "<password>", mode: "requireSSL" } }, replication: { oplogSizeMB: 1024, replSet: "juju" }, security: { authorization: "enabled", keyFile: "/var/lib/juju/shared-secret" }, storage: { dbPath: "/var/lib/juju/db", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "syslog", quiet: true } }
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] exception in initAndListen: 28662 Cannot start server. Detected data files in /var/lib/juju/db created by the 'mmapv1' storage engine, but the specified storage engine was 'wiredTiger'., terminating
Jan 17 22:37:21 juju-3bc6d2-controller-0 mongod.37017[1519]: [initandlisten] dbexit: rc: 100
Jan 17 22:37:21 juju-3bc6d2-controller-0 systemd[1]: juju-db.service: Main process exited, code=exited, status=100/n/a
Jan 17 22:37:21 juju-3bc6d2-controller-0 systemd[1]: juju-db.service: Unit entered failed state.
Jan 17 22:37:21 juju-3bc6d2-controller-0 systemd[1]: juju-db.service: Failed with result 'exit-code'.
Jan 17 22:37:22 juju-3bc6d2-controller-0 systemd[1]: juju-db.service: Service hold-off time over, scheduling restart.
Jan 17 22:37:22 juju-3bc6d2-controller-0 systemd[1]: Stopped juju state database.
Jan 17 22:37:22 juju-3bc6d2-controller-0 systemd[1]: juju-db.service: Start request repeated too quickly.
Jan 17 22:37:22 juju-3bc6d2-controller-0 systemd[1]: Failed to start juju state database.
This is an interesting part (and probably a separate bug):
2017-04-14 16:17:51 INFO juju.utils. packaging. manager utils.go:98 Retrying: &{/usr/bin/apt-get [apt-get --option= Dpkg::Options: :=--force- confold --option= Dpkg::options: :=--force- unsafe- io --ass chrono1. 58.0 libboost- regex1. 58.0 libyaml-cpp0.5v5 chrono1. 58.0 libboost- regex1. 58.0 libyaml-cpp0.5v5 nova.clouds. archive. ubuntu. com/ubuntu xenial- updates/ universe amd64 juju-mongodb3.2 amd64 3.2.9-0ubuntu1~ 16.04 nova.clouds. archive. ubuntu. com/ubuntu/ pool/universe/ j/juju- mongodb3. 2/juju- mongodb3. 2_3.2.9- 0ubuntu1~ 16.04_amd64. deb 404 Not Found [IP: 91.189.92.201 80]
ume-yes --quiet install juju-mongodb3.2] [] <nil> Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
libboost-
The following NEW packages will be installed:
juju-mongodb3.2 libboost-
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 24.4 MB/24.8 MB of archives.
After this operation, 86.9 MB of additional disk space will be used.
Err:1 http://
404 Not Found [IP: 91.189.92.201 80]
E: Failed to fetch http://
Does juju not do apt-get update before trying to install?