ceilometer-api fails to start on xenial-newton (maas + lxd)

[Ryan Beisner]

ceilometer-api fails to start on xenial-newton (bare metal tests):

ceilometer/0 blocked idle 1.25.6.1 4/lxc/0 8777/tcp 10.245.169.194 Services not running that should be: ceilometer-api

# juju status:
http://paste.ubuntu.com/23315659/

# system service status:
ubuntu@juju-machine-4-lxc-0:/etc/init$ sudo service ceilometer-api status
sudo: unable to resolve host juju-machine-4-lxc-0
● ceilometer-api.service - Ceilometer API
   Loaded: loaded (/lib/systemd/system/ceilometer-api.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/ceilometer-api.service.d
           └─override.conf
   Active: inactive (dead) (Result: exit-code) since Thu 2016-10-13 01:26:18 UTC; 37min ago
  Process: 21012 ExecStart=/usr/bin/ceilometer-api --port ${PORT} -- --config-file=/etc/ceilometer/ceilometer.conf --log-file=/var/log/ceilometer/ceilometer-api.log (code=exited, status=1/FAILURE)
  Process: 21011 ExecStartPre=/bin/chown ceilometer:adm /var/log/ceilometer (code=exited, status=0/SUCCESS)
  Process: 21010 ExecStartPre=/bin/chown ceilometer:ceilometer /var/lock/ceilometer /var/lib/ceilometer (code=exited, status=0/SUCCESS)
  Process: 21009 ExecStartPre=/bin/mkdir -p /var/lock/ceilometer /var/log/ceilometer /var/lib/ceilometer (code=exited, status=0/SUCCESS)
 Main PID: 21012 (code=exited, status=1/FAILURE)

Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: ceilometer-api.service: Unit entered failed state.
Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: ceilometer-api.service: Failed with result 'exit-code'.
Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: ceilometer-api.service: Service hold-off time over, scheduling restart.
Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: Stopped Ceilometer API.
Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: ceilometer-api.service: Start request repeated too quickly.
Oct 13 01:26:18 juju-machine-4-lxc-0 systemd[1]: Failed to start Ceilometer API.
Warning: ceilometer-api.service changed on disk. Run 'systemctl daemon-reload' to reload units.

[James Page (james-page)]

From ceilometer log output (in syslog):

Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager [-] Unable to discover resources: 'NoneType' object has no attribute 'get_access'
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager Traceback (most recent call last):
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 491, in discover
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager self.keystone).get_endpoints(
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/keystone_client.py", line 41, in get_service_catalog
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager return client.session.auth.get_access(client.session).service_catalog
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager AttributeError: 'NoneType' object has no attribute 'get_access'
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.163 21448 ERROR ceilometer.agent.manager
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager [-] Unable to discover resources: 'NoneType' object has no attribute 'get_access'
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager Traceback (most recent call last):
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/agent/manager.py", line 491, in discover
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager self.keystone).get_endpoints(
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager File "/usr/lib/python2.7/dist-packages/ceilometer/keystone_client.py", line 41, in get_service_catalog
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager return client.session.auth.get_access(client.session).service_catalog
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager AttributeError: 'NoneType' object has no attribute 'get_access'
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31.164 21448 ERROR ceilometer.agent.manager
Oct 6 12:56:31 juju-machine-4-lxc-0 ceilometer-agent-central[21435]: 2016-10-06 12:56:31...

[James Page (james-page)]

This looks wrong:

[service_credentials]
os_auth_url = http://10.245.169.120:5000/v2.0
os_tenant_name = services
os_username = ceilometer
os_password = kmMx7rp27XRFnrhFp2MPVBZ5bGZhgSrd6wrSSWYbyzX97VtmR4WSw5bjrcb8RLjj

[James Page (james-page)]

With the latest master branch charm:

[service_credentials]
auth_url = http://10.5.21.55:5000
project_name = services
username = ceilometer
password = zXMcH7kPzL8YGjFxsp8SPKckxhTCbkWcVCCJ5d4fq7KZWdtHJp6jCCcGh5bnC8Ys
project_domain_name = default
user_domain_name = default
auth_type = password

is what I see testing using newton

[Ryan Beisner (1chb1n)]

# mojo deploy output shows charm store retrieval
00:12:58.903 2016-10-13 10:29:38 [INFO] Downloading ~openstack-charmers-next/ceilometer-agent from charm store to ceilometer-agent
00:12:58.906 2016-10-13 10:29:38 [INFO] Downloading ~openstack-charmers-next/ceilometer from charm store to ceilometer
00:12:58.908 2016-10-13 10:29:38 [INFO] Downloading ~openstack-charmers-next/aodh from charm store to aodh
00:13:00.229 2016-10-13 10:29:39 [INFO] ceph-osd is cs:~openstack-charmers-next/ceph-osd-252
00:13:00.299 2016-10-13 10:29:39 [INFO] ntp is cs:trusty/ntp-15
00:13:00.322 2016-10-13 10:29:39 [INFO] ceilometer-agent is cs:~openstack-charmers-next/ceilometer-agent-230
00:13:00.678 2016-10-13 10:29:40 [INFO] mongodb is cs:trusty/mongodb-37
00:13:01.054 2016-10-13 10:29:40 [INFO] neutron-openvswitch is cs:~openstack-charmers-next/neutron-openvswitch-247
00:13:01.111 2016-10-13 10:29:40 [INFO] ceilometer is cs:~openstack-charmers-next/ceilometer-239
00:13:01.113 2016-10-13 10:29:40 [INFO] cinder-ceph is cs:~openstack-charmers-next/cinder-ceph-226

# repo-info from mojo charm collect dir shows the retrieved charm commit f289429
http://paste.ubuntu.com/23317381/

==> ./ceilometer/repo-info <==
commit-sha-1: f289429ed66138f1786bb89d02cf6a4aec476241
commit-short: f289429
branch: HEAD
remote: https://github.com/openstack/charm-ceilometer
info-generated: Wed Oct 12 07:16:14 UTC 2016
note: This file should exist only in a built or published charm artifact (not in the charm source code tree).

# repo-info on disk from the deployed unit shows charm commit f289429

jenkins@juju-osci1-machine-4:~$ juju ssh ceilometer/0 "cat /var/lib/juju/agents/unit-ceilometer-0/charm/repo-info"
Warning: Permanently added '10.245.168.11' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.245.170.170' (ECDSA) to the list of known hosts.
commit-sha-1: f289429ed66138f1786bb89d02cf6a4aec476241
commit-short: f289429
branch: HEAD
remote: https://github.com/openstack/charm-ceilometer
info-generated: Wed Oct 12 07:16:14 UTC 2016
note: This file should exist only in a built or published charm artifact (not in the charm source code tree).
Connection to 10.245.170.170 closed.

# git master shows current commet level f289429

'Stop/Start ceilometer-api if Apache has changed'
https://github.com/openstack/charm-ceilometer/commit/f289429ed66138f1786bb89d02cf6a4aec476241

[Ryan Beisner (1chb1n)]

This is ultimately caused by LXD containers using the MAAS provider not being able to resolve one-anothers' hostnames. Juju 1.25.6 + MAAS 1.9.4

[Ryan Beisner (1chb1n)]

<jamespage> Oct 13 10:54:41 juju-machine-4-lxc-0 ceilometer-api[18098]: my_ip = socket.gethostbyname(socket.gethostname())
<jamespage> Oct 13 10:54:41 juju-machine-4-lxc-0 ceilometer-api[18098]: socket.gaierror: [Errno -2] Name or service not known

[Anastasia (anastasia-macmood)]

1.25 on xenial does not support the lxd and newer lxc there. LXD work is in 2.0 only.

Marking this as "Won't Fix".

[Anastasia (anastasia-macmood)]

@Ryan,
Do you still see this failure with Juju 2.1?

[Ryan Beisner (1chb1n)]

This is still impacting Xenial-Newton Ceilometer on metal mojo spec testing.

[Ryan Beisner (1chb1n)]

xenial-newton ceilometer-api sees this fairly often. I do not observe it on any other combo in the OpenStack bare metal mojo spec tests.

[Tim Penhey (thumper)]

@Ryan we don't have anyone working on this just now, and I'm going to stop pretending that someone will just pick this up and work on it.

If this is still a blocking issue, we should get it onto the stakeholders kanban board to prioritise with other blocking issues.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.