rabbitmq-server some time fails to start on install leading to install hook failure

[Frode Nordahl]

2018-04-16 14:08:29 DEBUG install Adding group `rabbitmq' (GID 117) ...
2018-04-16 14:08:29 DEBUG install Done.
2018-04-16 14:08:29 DEBUG install Adding system user `rabbitmq' (UID 113) ...
2018-04-16 14:08:29 DEBUG install Adding new user `rabbitmq' (UID 113) with group `rabbitmq' ...
2018-04-16 14:08:29 DEBUG install Not creating home directory `/var/lib/rabbitmq'.
2018-04-16 14:08:34 DEBUG install Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
2018-04-16 14:08:34 DEBUG install invoke-rc.d: initscript rabbitmq-server, action "start" failed.
2018-04-16 14:08:34 DEBUG install ● rabbitmq-server.service - RabbitMQ Messaging Server
2018-04-16 14:08:34 DEBUG install Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
2018-04-16 14:08:34 DEBUG install Active: failed (Result: exit-code) since Mon 2018-04-16 14:08:34 UTC; 35ms ago
2018-04-16 14:08:34 DEBUG install Process: 13542 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=2)
2018-04-16 14:08:34 DEBUG install Process: 13537 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
2018-04-16 14:08:34 DEBUG install Main PID: 13537 (code=exited, status=1/FAILURE)
2018-04-16 14:08:34 DEBUG install
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:30 juju-5d893d-auto-osci-sv06-59 systemd[1]: Starting RabbitMQ M...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:31 juju-5d893d-auto-osci-sv06-59 rabbitmq[13542]: Waiting for 'r...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:31 juju-5d893d-auto-osci-sv06-59 rabbitmq[13542]: pid is 13545 ...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 rabbitmq[13542]: Error: process...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 systemd[1]: Failed to start Rab...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:34 DEBUG install Apr 16 14:08:34 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:34 DEBUG install Hint: Some lines were ellipsized, use -l to show in full.
2018-04-16 14:08:34 DEBUG install dpkg: error processing package rabbitmq-server (--configure):
2018-04-16 14:08:34 DEBUG install subprocess installed post-installation script returned error exit status 1
2018-04-16 14:08:34 DEBUG install Processing triggers for libc-bin (2.23-0ubuntu10) ...
2018-04-16 14:08:34 DEBUG install Processing triggers for systemd (229-4ubuntu21.2) ...
2018-04-16 14:08:35 DEBUG install Processing triggers for ureadahead (0.100.0-19) ...
2018-04-16 14:08:35 DEBUG install Errors were encountered while processing:
2018-04-16 14:08:35 DEBUG install rabbitmq-server
2018-04-16 14:08:38 DEBUG install E: Sub-process /usr/bin/dpkg returned an error code (1)
2018-04-16 14:08:38 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
2018-04-16 14:08:48 DEBUG install Reading package lists...
2018-04-16 14:08:48 DEBUG install Building dependency tree...
2018-04-16 14:08:48 DEBUG install Reading state information...
2018-04-16 14:08:49 DEBUG install lockfile-progs is already the newest version (0.1.17).
2018-04-16 14:08:49 DEBUG install python-amqplib is already the newest version (1.0.2-1).
2018-04-16 14:08:49 DEBUG install rabbitmq-server is already the newest version (3.5.7-1ubuntu0.16.04.2).
2018-04-16 14:08:49 DEBUG install 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
2018-04-16 14:08:49 DEBUG install 1 not fully installed or removed.
2018-04-16 14:08:49 DEBUG install After this operation, 0 B of additional disk space will be used.
2018-04-16 14:08:49 DEBUG install Setting up rabbitmq-server (3.5.7-1ubuntu0.16.04.2) ...
2018-04-16 14:08:54 DEBUG install Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
2018-04-16 14:08:54 DEBUG install invoke-rc.d: initscript rabbitmq-server, action "start" failed.
2018-04-16 14:08:54 DEBUG install ● rabbitmq-server.service - RabbitMQ Messaging Server
2018-04-16 14:08:54 DEBUG install Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
2018-04-16 14:08:54 DEBUG install Active: failed (Result: exit-code) since Mon 2018-04-16 14:08:54 UTC; 15ms ago
2018-04-16 14:08:54 DEBUG install Process: 13856 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=2)
2018-04-16 14:08:54 DEBUG install Process: 13853 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
2018-04-16 14:08:54 DEBUG install Main PID: 13853 (code=exited, status=1/FAILURE)
2018-04-16 14:08:54 DEBUG install
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:50 juju-5d893d-auto-osci-sv06-59 systemd[1]: Starting RabbitMQ M...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:51 juju-5d893d-auto-osci-sv06-59 rabbitmq[13856]: Waiting for 'r...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:51 juju-5d893d-auto-osci-sv06-59 rabbitmq[13856]: pid is 13871 ...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 rabbitmq[13856]: Error: process...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 systemd[1]: Failed to start Rab...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:54 DEBUG install Apr 16 14:08:54 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:08:54 DEBUG install Hint: Some lines were ellipsized, use -l to show in full.
2018-04-16 14:08:54 DEBUG install dpkg: error processing package rabbitmq-server (--configure):
2018-04-16 14:08:54 DEBUG install subprocess installed post-installation script returned error exit status 1
2018-04-16 14:08:54 DEBUG install Errors were encountered while processing:
2018-04-16 14:08:54 DEBUG install rabbitmq-server
2018-04-16 14:08:57 DEBUG install E: Sub-process /usr/bin/dpkg returned an error code (1)
2018-04-16 14:08:57 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
2018-04-16 14:09:07 DEBUG install Reading package lists...
2018-04-16 14:09:08 DEBUG install Building dependency tree...
2018-04-16 14:09:08 DEBUG install Reading state information...
2018-04-16 14:09:09 DEBUG install lockfile-progs is already the newest version (0.1.17).
2018-04-16 14:09:09 DEBUG install python-amqplib is already the newest version (1.0.2-1).
2018-04-16 14:09:09 DEBUG install rabbitmq-server is already the newest version (3.5.7-1ubuntu0.16.04.2).
2018-04-16 14:09:09 DEBUG install 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
2018-04-16 14:09:09 DEBUG install 1 not fully installed or removed.
2018-04-16 14:09:09 DEBUG install After this operation, 0 B of additional disk space will be used.
2018-04-16 14:09:09 DEBUG install Setting up rabbitmq-server (3.5.7-1ubuntu0.16.04.2) ...
2018-04-16 14:09:14 DEBUG install Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
2018-04-16 14:09:14 DEBUG install invoke-rc.d: initscript rabbitmq-server, action "start" failed.
2018-04-16 14:09:14 DEBUG install ● rabbitmq-server.service - RabbitMQ Messaging Server
2018-04-16 14:09:14 DEBUG install Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
2018-04-16 14:09:14 DEBUG install Active: failed (Result: exit-code) since Mon 2018-04-16 14:09:14 UTC; 25ms ago
2018-04-16 14:09:14 DEBUG install Process: 14144 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=2)
2018-04-16 14:09:14 DEBUG install Process: 14138 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
2018-04-16 14:09:14 DEBUG install Main PID: 14138 (code=exited, status=1/FAILURE)
2018-04-16 14:09:14 DEBUG install
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:10 juju-5d893d-auto-osci-sv06-59 systemd[1]: Starting RabbitMQ M...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:10 juju-5d893d-auto-osci-sv06-59 rabbitmq[14144]: Waiting for 'r...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:10 juju-5d893d-auto-osci-sv06-59 rabbitmq[14144]: pid is 14145 ...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:13 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:14 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:14 juju-5d893d-auto-osci-sv06-59 systemd[1]: Failed to start Rab...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:14 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:14 DEBUG install Apr 16 14:09:14 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:14 DEBUG install Hint: Some lines were ellipsized, use -l to show in full.
2018-04-16 14:09:14 DEBUG install dpkg: error processing package rabbitmq-server (--configure):
2018-04-16 14:09:14 DEBUG install subprocess installed post-installation script returned error exit status 1
2018-04-16 14:09:14 DEBUG install Errors were encountered while processing:
2018-04-16 14:09:14 DEBUG install rabbitmq-server
2018-04-16 14:09:16 DEBUG install E: Sub-process /usr/bin/dpkg returned an error code (1)
2018-04-16 14:09:16 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds
2018-04-16 14:09:27 DEBUG install Reading package lists...
2018-04-16 14:09:27 DEBUG install Building dependency tree...
2018-04-16 14:09:27 DEBUG install Reading state information...
2018-04-16 14:09:28 DEBUG install lockfile-progs is already the newest version (0.1.17).
2018-04-16 14:09:28 DEBUG install python-amqplib is already the newest version (1.0.2-1).
2018-04-16 14:09:28 DEBUG install rabbitmq-server is already the newest version (3.5.7-1ubuntu0.16.04.2).
2018-04-16 14:09:28 DEBUG install 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
2018-04-16 14:09:28 DEBUG install 1 not fully installed or removed.
2018-04-16 14:09:28 DEBUG install After this operation, 0 B of additional disk space will be used.
2018-04-16 14:09:28 DEBUG install Setting up rabbitmq-server (3.5.7-1ubuntu0.16.04.2) ...
2018-04-16 14:09:33 DEBUG install Job for rabbitmq-server.service failed because the control process exited with error code. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
2018-04-16 14:09:33 DEBUG install invoke-rc.d: initscript rabbitmq-server, action "start" failed.
2018-04-16 14:09:33 DEBUG install ● rabbitmq-server.service - RabbitMQ Messaging Server
2018-04-16 14:09:33 DEBUG install Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
2018-04-16 14:09:33 DEBUG install Active: failed (Result: exit-code) since Mon 2018-04-16 14:09:33 UTC; 29ms ago
2018-04-16 14:09:33 DEBUG install Process: 14431 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=2)
2018-04-16 14:09:33 DEBUG install Process: 14427 ExecStart=/usr/sbin/rabbitmq-server (code=exited, status=1/FAILURE)
2018-04-16 14:09:33 DEBUG install Main PID: 14427 (code=exited, status=1/FAILURE)
2018-04-16 14:09:33 DEBUG install
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:29 juju-5d893d-auto-osci-sv06-59 systemd[1]: Starting RabbitMQ M...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:30 juju-5d893d-auto-osci-sv06-59 rabbitmq[14431]: Waiting for 'r...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:30 juju-5d893d-auto-osci-sv06-59 rabbitmq[14431]: pid is 14434 ...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:32 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:33 juju-5d893d-auto-osci-sv06-59 rabbitmq[14431]: Error: process...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:33 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:33 juju-5d893d-auto-osci-sv06-59 systemd[1]: Failed to start Rab...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:33 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:33 DEBUG install Apr 16 14:09:33 juju-5d893d-auto-osci-sv06-59 systemd[1]: rabbitmq-server.ser...
2018-04-16 14:09:33 DEBUG install Hint: Some lines were ellipsized, use -l to show in full.
2018-04-16 14:09:33 DEBUG install dpkg: error processing package rabbitmq-server (--configure):
2018-04-16 14:09:33 DEBUG install subprocess installed post-installation script returned error exit status 1
2018-04-16 14:09:33 DEBUG install Errors were encountered while processing:
2018-04-16 14:09:33 DEBUG install rabbitmq-server
2018-04-16 14:09:36 DEBUG install E: Sub-process /usr/bin/dpkg returned an error code (1)
2018-04-16 14:09:36 DEBUG install Traceback (most recent call last):
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/install.real", line 795, in <module>
2018-04-16 14:09:36 DEBUG install hooks.execute(sys.argv)
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-04-16 14:09:36 DEBUG install self._hooks[hook_name]()
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2018-04-16 14:09:36 DEBUG install return f(*args, **kwargs)
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/install.real", line 122, in install
2018-04-16 14:09:36 DEBUG install rabbit.install_or_upgrade_packages()
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/rabbit_utils.py", line 1147, in install_or_upgrade_packages
2018-04-16 14:09:36 DEBUG install apt_install(PACKAGES, fatal=True)
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/charmhelpers/fetch/ubuntu.py", line 207, in apt_install
2018-04-16 14:09:36 DEBUG install _run_apt_command(cmd, fatal)
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/charmhelpers/fetch/ubuntu.py", line 559, in _run_apt_command
2018-04-16 14:09:36 DEBUG install retry_message="Couldn't acquire DPKG lock")
2018-04-16 14:09:36 DEBUG install File "/var/lib/juju/agents/unit-rabbitmq-server-4/charm/hooks/charmhelpers/fetch/ubuntu.py", line 535, in _run_with_retries
2018-04-16 14:09:36 DEBUG install result = subprocess.check_call(cmd, **kwargs)
2018-04-16 14:09:36 DEBUG install File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
2018-04-16 14:09:36 DEBUG install raise CalledProcessError(retcode, cmd)
2018-04-16 14:09:36 DEBUG install subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option=Dpkg::Options::=--force-confold', 'install', 'rabbitmq-server', 'python-amqplib', 'lockfile-progs']' returned non-zero exit status 100
2018-04-16 14:09:36 ERROR juju.worker.uniter.operation runhook.go:113 hook "install" failed: exit status 1

$ cat rabbitmq/startup_log
ERROR: epmd error for host juju-5d893d-auto-osci-sv06-59: address (cannot connect to host/port)

More information available here:
https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_amulet_full/openstack/charm-ceph-osd/561565/1/1355/index.html