Activity log for bug #1645849

Date Who What changed Old value New value Message
2016-11-29 20:34:47 James Pham bug added bug
2016-11-29 20:50:05 James Pham description Hi, the "kolla-ansible deploy" process fails at the following step: TASK [common : Creating log volume] ******************************************** ok: [localhost] TASK [common : Starting heka container] **************************************** changed: [localhost] TASK [common : Starting kolla-toolbox container] ******************************* fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_Rc40VA/ansible_module_kolla_docker.py\", line 74 2, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_Rc40VA/ansible_module_kolla_docker.py\", line 572, in start_container\\n se lf.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *a rgs, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 363, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-package s/docker/client.py\", line 146, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"Cannot start container ab9ea9073bd35e491393cb85813e980c0e93fad2796518375cf8468945000f4e: [9] System error: invalid character \\'r\\' looking for beginning of value\")\\n'"} Running the "docker ps" command gives (heka is the only container that was started and it keeps on restarting: [root@localhost heka]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 09aec584bc73 kolla/centos-binary-heka:3.0.1 "kolla_start" 13 minutes ago Restarting (1) 11 seconds ago heka Running "docker logs heka" gives the following "/var/cache/hekad Permission Denied: [root@localhost heka]# docker logs heka INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:/var/lib/kolla/config_files/heka-elasticsearch.toml does not exist, but is not required INFO:__main__:Copying /var/lib/kolla/config_files/heka-global.toml to /etc/heka/heka-global.toml INFO:__main__:Setting permissions for /etc/heka/heka-global.toml INFO:__main__:/var/lib/kolla/config_files/heka-aodh.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-gnocchi.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-barbican.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-custom.toml does not exist, but is not required INFO:__main__:Copying /var/lib/kolla/config_files/heka-haproxy.toml to /etc/heka/heka-haproxy.toml INFO:__main__:Setting permissions for /etc/heka/heka-haproxy.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-keepalived.toml to /etc/heka/heka-keepalived.toml INFO:__main__:Setting permissions for /etc/heka/heka-keepalived.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-rabbitmq.toml to /etc/heka/heka-rabbitmq.toml INFO:__main__:Setting permissions for /etc/heka/heka-rabbitmq.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-openstack.toml to /etc/heka/heka-openstack.toml INFO:__main__:Setting permissions for /etc/heka/heka-openstack.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-mariadb.toml to /etc/heka/heka-mariadb.toml INFO:__main__:Setting permissions for /etc/heka/heka-mariadb.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-keystone.toml to /etc/heka/heka-keystone.toml INFO:__main__:Setting permissions for /etc/heka/heka-keystone.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-horizon.toml to /etc/heka/heka-horizon.toml INFO:__main__:Setting permissions for /etc/heka/heka-horizon.toml INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-reaper.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-updater.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-expirer.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-updater.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-proxy-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-rsyncd.toml does not exist, but is not required INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/cache/hekad ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 304, in <module> exit_code = main() File "/usr/local/bin/kolla_set_configs", line 298, in main execute_config_strategy() File "/usr/local/bin/kolla_set_configs", line 235, in execute_config_strategy handle_permissions(config) File "/usr/local/bin/kolla_set_configs", line 219, in handle_permissions set_perms(dest, uid, gid) File "/usr/local/bin/kolla_set_configs", line 216, in set_perms os.chown(path, uid, gid) OSError: [Errno 13] Permission denied: '/var/cache/hekad' Using Kolla 3.0.1 on a CentOS 7 machine. Hi, the "kolla-ansible deploy" process fails at the following step: TASK [common : Creating log volume] ******************************************** ok: [localhost] TASK [common : Starting heka container] **************************************** changed: [localhost] TASK [common : Starting kolla-toolbox container] ******************************* fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\\n File \"/tmp/ansible_Rc40VA/ansible_module_kolla_docker.py\", line 74 2, in main\\n result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n File \"/tmp/ansible_Rc40VA/ansible_module_kolla_docker.py\", line 572, in start_container\\n se lf.dc.start(container=self.params.get(\\'name\\'))\\n File \"/usr/lib/python2.7/site-packages/docker/utils/decorators.py\", line 21, in wrapped\\n return f(self, resource_id, *a rgs, **kwargs)\\n File \"/usr/lib/python2.7/site-packages/docker/api/container.py\", line 363, in start\\n self._raise_for_status(res)\\n File \"/usr/lib/python2.7/site-package s/docker/client.py\", line 146, in _raise_for_status\\n raise errors.APIError(e, response, explanation=explanation)\\nAPIError: 500 Server Error: Internal Server Error (\"Cannot start container ab9ea9073bd35e491393cb85813e980c0e93fad2796518375cf8468945000f4e: [9] System error: invalid character \\'r\\' looking for beginning of value\")\\n'"} Running the "docker ps" command gives (heka is the only container that was started and it keeps on restarting: [root@localhost heka]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 09aec584bc73 kolla/centos-binary-heka:3.0.1 "kolla_start" 13 minutes ago Restarting (1) 11 seconds ago heka Running "docker logs heka" gives the following "/var/cache/hekad Permission Denied: [root@localhost heka]# docker logs heka INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Copying service configuration files INFO:__main__:/var/lib/kolla/config_files/heka-elasticsearch.toml does not exist, but is not required INFO:__main__:Copying /var/lib/kolla/config_files/heka-global.toml to /etc/heka/heka-global.toml INFO:__main__:Setting permissions for /etc/heka/heka-global.toml INFO:__main__:/var/lib/kolla/config_files/heka-aodh.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-gnocchi.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-barbican.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-custom.toml does not exist, but is not required INFO:__main__:Copying /var/lib/kolla/config_files/heka-haproxy.toml to /etc/heka/heka-haproxy.toml INFO:__main__:Setting permissions for /etc/heka/heka-haproxy.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-keepalived.toml to /etc/heka/heka-keepalived.toml INFO:__main__:Setting permissions for /etc/heka/heka-keepalived.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-rabbitmq.toml to /etc/heka/heka-rabbitmq.toml INFO:__main__:Setting permissions for /etc/heka/heka-rabbitmq.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-openstack.toml to /etc/heka/heka-openstack.toml INFO:__main__:Setting permissions for /etc/heka/heka-openstack.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-mariadb.toml to /etc/heka/heka-mariadb.toml INFO:__main__:Setting permissions for /etc/heka/heka-mariadb.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-keystone.toml to /etc/heka/heka-keystone.toml INFO:__main__:Setting permissions for /etc/heka/heka-keystone.toml INFO:__main__:Copying /var/lib/kolla/config_files/heka-horizon.toml to /etc/heka/heka-horizon.toml INFO:__main__:Setting permissions for /etc/heka/heka-horizon.toml INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-reaper.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-account-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-container-updater.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-auditor.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-expirer.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-replicator.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-object-updater.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-proxy-server.toml does not exist, but is not required INFO:__main__:/var/lib/kolla/config_files/heka-swift-rsyncd.toml does not exist, but is not required INFO:__main__:Writing out command to execute INFO:__main__:Setting permission for /var/cache/hekad ERROR:__main__:Unexpected error: Traceback (most recent call last):   File "/usr/local/bin/kolla_set_configs", line 304, in <module>     exit_code = main()   File "/usr/local/bin/kolla_set_configs", line 298, in main     execute_config_strategy()   File "/usr/local/bin/kolla_set_configs", line 235, in execute_config_strategy     handle_permissions(config)   File "/usr/local/bin/kolla_set_configs", line 219, in handle_permissions     set_perms(dest, uid, gid)   File "/usr/local/bin/kolla_set_configs", line 216, in set_perms     os.chown(path, uid, gid) OSError: [Errno 13] Permission denied: '/var/cache/hekad'
2016-12-01 14:02:20 Paul Bourke kolla: status New Triaged
2016-12-16 05:21:41 Jeffrey Zhang kolla: milestone ocata-3
2016-12-16 14:33:14 Christian Berendt kolla: importance Undecided Low
2017-01-29 23:34:40 Jeffrey Zhang kolla: milestone ocata-3 ocata-rc1
2017-02-22 07:55:13 Jeffrey Zhang kolla: status Triaged Won't Fix