[AODH] Failed to run stop playbook

Bug #1864376 reported by Viorel-Cosmin Miron
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Invalid
Undecided
Unassigned

Bug Description

Tried first to run kolla-ansible destroy this resulted in: fatal:

[sc]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "Shared connection to sc closed.\r\n", "stderr_lines": ["Shared connection to sc closed."], "stdout": "Some qemu processes were detected.\r\nDocker will not be able to stop the nova_libvirt container with those running.\r\nPlease clean them up before rerunning this script.\r\n", "stdout_lines": ["Some qemu processes were detected.", "Docker will not be able to stop the nova_libvirt container with those running.", "Please clean them up before rerunning this script."]}

After this I have runned kolla-ansinle stop and this resulted in:

TASK [aodh : include_tasks] *****************************************************************************************************************************************
fatal: [sc]: FAILED! => {"reason": "Could not find or access '/root/venv3/share/kolla-ansible/ansible/stop.yml' on the Ansible Controller."}

TASK [service-stop : Stopping common containers] ********************************************************************************************************************
failed: [ss] (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'enabled': True, 'image': 'kolla/ubuntu-source-fluentd:train', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "fluentd", "value": {"container_name": "fluentd", "dimensions": {}, "enabled": true, "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS"}, "image": "kolla/ubuntu-source-fluentd:train", "volumes": ["/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "kolla_logs:/var/log/kolla/", "fluentd_data:/var/lib/fluentd/data/"]}}, "msg": "No such container: fluentd to stop"}
failed: [ss] (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'enabled': True, 'image': 'kolla/ubuntu-source-kolla-toolbox:train', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "kolla-toolbox", "value": {"container_name": "kolla_toolbox", "dimensions": {}, "enabled": true, "environment": {"ANSIBLE_LIBRARY": "/usr/share/ansible", "ANSIBLE_NOCOLOR": "1"}, "image": "kolla/ubuntu-source-kolla-toolbox:train", "privileged": true, "volumes": ["/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "/dev/:/dev/", "/run/:/run/:shared", "kolla_logs:/var/log/kolla/"]}}, "msg": "No such container: kolla_toolbox to stop"}
failed: [ss] (item={'key': 'cron', 'value': {'container_name': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:train', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "cron", "value": {"container_name": "cron", "dimensions": {}, "enabled": true, "environment": {"DUMMY_ENVIRONMENT": "kolla_useless_env"}, "image": "kolla/ubuntu-source-cron:train", "volumes": ["/etc/kolla/cron/:/var/lib/kolla/config_files/:ro", "/etc/localtime:/etc/localtime:ro", "kolla_logs:/var/log/kolla/"]}}, "msg": "No such container: cron to stop"}
changed: [sc] => (item={'key': 'fluentd', 'value': {'container_name': 'fluentd', 'enabled': True, 'image': 'kolla/ubuntu-source-fluentd:train', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS'}, 'volumes': ['/etc/kolla/fluentd/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/', 'fluentd_data:/var/lib/fluentd/data/'], 'dimensions': {}}})
changed: [sc] => (item={'key': 'kolla-toolbox', 'value': {'container_name': 'kolla_toolbox', 'enabled': True, 'image': 'kolla/ubuntu-source-kolla-toolbox:train', 'environment': {'ANSIBLE_NOCOLOR': '1', 'ANSIBLE_LIBRARY': '/usr/share/ansible'}, 'privileged': True, 'volumes': ['/etc/kolla/kolla-toolbox/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', '/dev/:/dev/', '/run/:/run/:shared', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}})
changed: [sc] => (item={'key': 'cron', 'value': {'container_name': 'cron', 'enabled': True, 'image': 'kolla/ubuntu-source-cron:train', 'environment': {'DUMMY_ENVIRONMENT': 'kolla_useless_env'}, 'volumes': ['/etc/kolla/cron/:/var/lib/kolla/config_files/:ro', '/etc/localtime:/etc/localtime:ro', 'kolla_logs:/var/log/kolla/'], 'dimensions': {}}})

After reruning destroy this get stuck for now, will post laters if any logs show up.

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

Added multinode and globals.yml

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :
Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

Update: running again destroy now will get it stuck for endless time.

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

http://paste.openstack.org/show/789904/ destroy logs in verbose mode.

Revision history for this message
Viorel-Cosmin Miron (uhl-hosting) wrote :

It seems in end destroy work, just takes a lot :)

Changed in kolla-ansible:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.