failed to bootstrap ceilometer container

[Andrew Liu]

Failed to deploy ceilometer containers following this bp https://blueprints.launchpad.net/kolla/+spec/ansible-ceilometer

[How to reproduce]
a) Check out the latest code of kolla, which should include this review: https://review.openstack.org/306858

b) Cherrypick this review: https://review.openstack.org/300574

c) Enabled the mongodb and ceilometer in the globals.yml file,

d) Run command: kolla-ansible deploy, then got this error:

TASK: [ceilometer | Running Ceilometer bootstrap container] *******************
failed: [localhost -> localhost] => {"changed": true, "failed": true}
msg: Container exited with non-zero return code

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/site.retry

localhost : ok=308 changed=3 unreachable=0 failed=1

Command failed ansible-playbook -i /usr/share/kolla/ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla/ansible/site.yml

e) Check the log of the bootstrap_ceilometer in the docker

[root@localhost tools]# docker logs bootstrap_ceilometer
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Removing existing destination: /etc/ceilometer/ceilometer.conf
INFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf
INFO:__main__:Setting permissions for /etc/ceilometer/ceilometer.conf
INFO:__main__:Writing out command to execute
Running command: 'ceilometer-api'
Value 'password-ceilometer-legacy' for '[service_credentials]/auth_type' is deprecated. And will be removed in Ceilometer 7.0. Use 'password' instead.
Traceback (most recent call last):
  File "/usr/bin/ceilometer-api", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/ceilometer/cmd/api.py", line 22, in main
    service.prepare_service()
  File "/usr/lib/python2.7/site-packages/ceilometer/service.py", line 87, in prepare_service
    log.setup(cfg.CONF, 'ceilometer')
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 254, in setup
    _setup_logging_from_conf(conf, product_name, version)
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 351, in _setup_logging_from_conf
    filelog = file_handler(logpath)
  File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 2] No such file or directory: '/var/log/kolla/ceilometer/ceilometer-api.log'

[Swapnil Kulkarni (coolsvap-deactivatedaccount)]

This is still an in progress review. I am not sure a bug would be accepted for in progress reviews. Please put your debug information as a comment in review.

[Andrew Liu (andrew-lhj)]

I have already discussed with the author of the bp, he agrees to create a bug for it.

[Hui Kang (huikang27)]

Hi, Andrew, my deployment with ubuntu source success for the bootstrap process [1].
Actually the bootstrap ceilometer container will create the log directory [2].

Could you help to run ubuntu source in your testbed? Thanks. - Hui

[1] http://paste.openstack.org/show/497255/
[2] https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-base/extend_start.sh#L3

[Andrew Liu (andrew-lhj)]

I got this error when deploy with ubuntu and source.

[root@localhost kolla]# docker logs bootstrap_ceilometer
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Copying service configuration files
INFO:__main__:Copying /var/lib/kolla/config_files/ceilometer.conf to /etc/ceilometer/ceilometer.conf
INFO:__main__:Setting permissions for /etc/ceilometer/ceilometer.conf
INFO:__main__:Writing out command to execute
/usr/local/bin/kolla_extend_start: line 6: ceilometer-manage: command not found

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/328581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/328581
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=0c71f2d8ea9fbb661b09c0a35c7b135596e9635c
Submitter: Jenkins
Branch: master

commit 0c71f2d8ea9fbb661b09c0a35c7b135596e9635c
Author: Hui Kang <email address hidden>
Date: Fri Jun 10 23:12:40 2016 -0400

    Fix missing ceilometer user in the sudoers list for binary type

    ceilometer user and extend_start.sh should be added to both source
    and binary type

    Change-Id: I149e27e31c6eb200b508e5b67a98edccc8d86962
    closes-Bug: #1582062

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 3.0.0.0b2 development milestone.