Bug #1672240 “destroy tool is incomplete " : Bugs : kolla

Revision history for this message

zhubingbing (zhubingbing) wrote on 2017-03-13:

#1

good ;

Changed in kolla:
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

Duong Ha-Quang (duonghq) wrote on 2017-03-13:

#2

You should change the title to reflect description better: this bug only cover mariadb component.

Revision history for this message

Steven Dake (sdake) wrote on 2017-03-13:

#3

This may cover all named volumes in general that are not cleaned up by the destroy feature.

zhubingbing (zhubingbing) on 2017-03-13

Changed in kolla:
assignee:	nobody → zhubingbing (zhubingbing)
milestone:	none → pike-1

Revision history for this message

Duong Ha-Quang (duonghq) wrote on 2017-03-13:

#4

@Steve: so we should update the description?

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-13:

#5

Download full text (4.4 KiB)

TASK [mariadb : include] *******************************************************
included: /opt/bgi-kolla/kolla-ansible/ansible/roles/mariadb/tasks/start.yml for kode0, kode1, kode2

TASK [mariadb : Starting mariadb container] ************************************
skipping: [kode0]
changed: [kode1]
changed: [kode2]

TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
ok: [kode0]
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
fatal: [kode1]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_f30bRe/ansible_module_wait_for.py\", line 540, in <module>\r\n main()\r\n File \"/tmp/ansible_f30bRe/ansible_module_wait_for.py\", line 481, in main\r\n response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
fatal: [kode2]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode2 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_PzGW_h/ansible_module_wait_for.py\", line 540, in <module>\r\n main()\r\n File \"/tmp/ansible_PzGW_h/ansible_module_wait_for.py\",...

TASK [mariadb : include] *******************************************************
included: /opt/bgi-kolla/kolla-ansible/ansible/roles/mariadb/tasks/start.yml for kode0, kode1, kode2

TASK [mariadb : Starting mariadb container] ************************************
skipping: [kode0]
changed: [kode1]
changed: [kode2]

TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
ok: [kode0]
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
fatal: [kode1]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_f30bRe/ansible_module_wait_for.py\", line 540, in <module>\r\n    main()\r\n  File \"/tmp/ansible_f30bRe/ansible_module_wait_for.py\", line 481, in main\r\n    response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
fatal: [kode2]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode2 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_PzGW_h/ansible_module_wait_for.py\", line 540, in <module>\r\n    main()\r\n  File \"/tmp/ansible_PzGW_h/ansible_module_wait_for.py\", line 481, in main\r\n    response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}

TASK [mariadb : include] *******************************************************
included: /opt/bgi-kolla/kolla-ansible/ansible/roles/mariadb/tasks/register.yml for kode0

TASK [mariadb : Creating haproxy mysql user] ***********************************
changed: [kode0]

TASK [mariadb : Cleaning up facts] *********************************************
ok: [kode0]

TASK [mariadb : include] *******************************************************
included: /opt/bgi-kolla/kolla-ansible/ansible/roles/mariadb/tasks/start.yml for kode0

TASK [mariadb : Starting mariadb container] ************************************
changed: [kode0]

TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
^C [ERROR]: User interrupted execution

Command failed ansible-playbook -i ../ansible/inventory/bgi-openstack -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e horizon_tag=4.0.0.2 -e action=deploy /opt/bgi-kolla/kolla-ansible/ansible/site.yml 
[root@kola bgi]#

Revision history for this message

zhubingbing (zhubingbing) wrote on 2017-03-13:

#6

can u run kolla-ansible destroy --yes, next run
kolla-ansbile prechecks
kolla-ansible deploy

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-13:

#7

now, the issue hasn't been reproduced , I remember I clean up all iptable rules which is left in last deployment.

Is it something about the issue ?

Revision history for this message

zhubingbing (zhubingbing) wrote on 2017-03-13:

#8

i ' cant reproduce this bug.

Changed in kolla:
status:	Confirmed → Invalid

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-17:

#9

now this time I reproduced the issue even after following comment #6.

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-17:

#10

operations Edit (2.2 MiB, text/plain)

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-17:

#11

Download full text (8.5 KiB)

[root@kode1 docker]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9df398bdc781 192.168.103.16:5000/bgi/centos-binary-mariadb:4.0.0.1 "kolla_start" 2 minutes ago Restarting (0) 4 seconds ago mariadb
bf5d97a8776a 192.168.103.16:5000/bgi/centos-binary-memcached:4.0.0.1 "kolla_start" 2 minutes ago Up 2 minutes memcached
0ea71f1067da 192.168.103.16:5000/bgi/centos-binary-kibana:4.0.0.1 "kolla_start" 2 minutes ago Up 2 minutes kibana
b7ed891f11eb 192.168.103.16:5000/bgi/centos-binary-elasticsearch:4.0.0.1 "kolla_start" 4 minutes ago Up 4 minutes elasticsearch
578467f2157e 192.168.103.16:5000/bgi/centos-binary-cron:4.0.0.1 "kolla_start" 4 minutes ago Up 4 minutes cron
d48b5b4ab9d5 192.168.103.16:5000/bgi/centos-binary-kolla-toolbox:4.0.0.1 "kolla_start" 4 minutes ago Up 4 minutes kolla_toolbox
1c6ecbcb451c 192.168.103.16:5000/bgi/centos-binary-fluentd:4.0.0.1 "kolla_start" 4 minutes ago Up 4 minutes fluentd
[root@kode1 docker]# cd /var/lib/docker/volumes/kolla_logs/_data/mariadb/
[root@kode1 mariadb]# ls
mariadb.log

[root@kode1 mariadb]# tail -n 100 mariadb.log
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:48:49 [Note] WSREP: (710c31d9, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:16 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:16 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:16 [ERROR] Aborting

170317 21:49:16 [Note] WSREP: Service disconnected.
170317 21:49:17 [Note] WSREP: Some threads may fail to exit.
170317 21:49:17 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:17 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:18 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/my...

[root@kode1 docker]# docker ps 
CONTAINER ID        IMAGE                                                         COMMAND             CREATED             STATUS                         PORTS               NAMES
9df398bdc781        192.168.103.16:5000/bgi/centos-binary-mariadb:4.0.0.1         "kolla_start"       2 minutes ago       Restarting (0) 4 seconds ago                       mariadb
bf5d97a8776a        192.168.103.16:5000/bgi/centos-binary-memcached:4.0.0.1       "kolla_start"       2 minutes ago       Up 2 minutes                                       memcached
0ea71f1067da        192.168.103.16:5000/bgi/centos-binary-kibana:4.0.0.1          "kolla_start"       2 minutes ago       Up 2 minutes                                       kibana
b7ed891f11eb        192.168.103.16:5000/bgi/centos-binary-elasticsearch:4.0.0.1   "kolla_start"       4 minutes ago       Up 4 minutes                                       elasticsearch
578467f2157e        192.168.103.16:5000/bgi/centos-binary-cron:4.0.0.1            "kolla_start"       4 minutes ago       Up 4 minutes                                       cron
d48b5b4ab9d5        192.168.103.16:5000/bgi/centos-binary-kolla-toolbox:4.0.0.1   "kolla_start"       4 minutes ago       Up 4 minutes                                       kolla_toolbox
1c6ecbcb451c        192.168.103.16:5000/bgi/centos-binary-fluentd:4.0.0.1         "kolla_start"       4 minutes ago       Up 4 minutes                                       fluentd
[root@kode1 docker]# cd /var/lib/docker/volumes/kolla_logs/_data/mariadb/
[root@kode1 mariadb]# ls
mariadb.log
 
[root@kode1 mariadb]# tail -n 100 mariadb.log
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:48:49 [Note] WSREP: (710c31d9, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:16 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:16 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:16 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:16 [ERROR] Aborting

170317 21:49:16 [Note] WSREP: Service disconnected.
170317 21:49:17 [Note] WSREP: Some threads may fail to exit.
170317 21:49:17 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:17 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:18 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.c0BcOk' --pid-file='/var/lib/mysql//kode1.genomics.cn-recover.pid'
nohup: ignoring input
170317 21:49:18 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 179 ...
170317 21:49:21 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: Read nil XID from storage engines, skipping position init
170317 21:49:21 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
170317 21:49:21 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 217 ...
170317 21:49:21 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <info@codership.com> loaded successfully.
170317 21:49:21 [Note] WSREP: CRC-32C: using hardware acceleration.
170317 21:49:21 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
170317 21:49:21 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.102.21; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.102.21:4567; gmcast.segment = 0; gmcast.version = 0; ist
170317 21:49:21 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
170317 21:49:21 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
170317 21:49:21 [Note] WSREP: wsrep_sst_grab()
170317 21:49:21 [Note] WSREP: Start replication
170317 21:49:21 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
170317 21:49:21 [Note] WSREP: protonet asio version 0
170317 21:49:21 [Note] WSREP: Using CRC-32C for message checksums.
170317 21:49:21 [Note] WSREP: backend: asio
170317 21:49:21 [Note] WSREP: gcomm thread scheduling priority set to other:0 
170317 21:49:21 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
170317 21:49:21 [Note] WSREP: restore pc from disk failed
170317 21:49:21 [Note] WSREP: GMCast version 0
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') listening at tcp://192.168.102.21:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') multicast: , ttl: 1
170317 21:49:21 [Note] WSREP: EVS version 0
170317 21:49:21 [Note] WSREP: gcomm: connecting to group 'openstack', peer '192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567'
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') connection established to 9fd9d50c tcp://192.168.102.22:4567
170317 21:49:21 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting on, nonlive peers: 
170317 21:49:21 [Note] WSREP: declaring 9fd9d50c at tcp://192.168.102.22:4567 stable
170317 21:49:21 [Note] WSREP: view(view_id(NON_PRIM,863108cf,40) memb {
        863108cf,0
        9fd9d50c,0
} joined {
} left {
} partitioned {
        0745c975,0
        1c699739,0
        319270e6,0
        46b83db9,0
        5bdeb2c7,0
        5dc7f26e,0
        710c31d9,0
        7338683d,0
        885fabba,0
        9d856ffe,0
        9eed86eb,0
        b2ae1580,0
        c7cff5ac,0
        dcf41ff9,0
        f21f343e,0
})
170317 21:49:25 [Note] WSREP: (863108cf, 'tcp://192.168.102.21:4567') turning message relay requesting off
170317 21:49:52 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170317 21:49:52 [ERROR] WSREP: gcs connect failed: Connection timed out
170317 21:49:52 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170317 21:49:52 [ERROR] Aborting

170317 21:49:52 [Note] WSREP: Service disconnected.
170317 21:49:53 [Note] WSREP: Some threads may fail to exit.
170317 21:49:53 [Note] /usr/sbin/mysqld: Shutdown complete

170317 21:49:53 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
170317 21:49:53 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/
170317 21:49:53 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql//wsrep_recovery.xhDg6X' --pid-file='/var/lib/mysql//kode1.genomics.cn-recover.pid'
nohup: ignoring input
170317 21:49:53 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-wsrep) starting as process 178 ...

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-18:

#12

Download full text (5.8 KiB)

I try to destroy with clean image then deploy the environment but still reproduced the issue.

this time :
TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
fatal: [kode1]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 540, in <module>\r\n main()\r\n File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 481, in main\r\n response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
fatal: [kode2]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared con...

I try to destroy with clean image then deploy the environment but still reproduced the issue.

this time :
TASK [mariadb : Waiting for MariaDB service to be ready] ***********************
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (10 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (9 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (8 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (7 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (6 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (5 retries left).
fatal: [kode1]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode1 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 540, in <module>\r\n    main()\r\n  File \"/tmp/ansible_Q9j2N1/ansible_module_wait_for.py\", line 481, in main\r\n    response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
fatal: [kode2]: FAILED! => {"attempts": 10, "changed": false, "failed": true, "module_stderr": "Shared connection to kode2 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_7uFJw5/ansible_module_wait_for.py\", line 540, in <module>\r\n    main()\r\n  File \"/tmp/ansible_7uFJw5/ansible_module_wait_for.py\", line 481, in main\r\n    response = s.recv(1024)\r\nsocket.error: [Errno 104] Connection reset by peer\r\n", "msg": "MODULE FAILURE"}
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (4 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (3 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (2 retries left).
FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready (1 retries left).
fatal: [kode0]: FAILED! => {"attempts": 10, "changed": false, "elapsed": 60, "failed": true, "msg": "Timeout when waiting for search string MariaDB in 192.168.102.20:3306"}
        to retry, use: --limit @/opt/bgi-kolla/kolla-ansible/ansible/site.retry

PLAY RECAP *********************************************************************
kode0                      : ok=62   changed=31   unreachable=0    failed=1   
kode1                      : ok=58   changed=32   unreachable=0    failed=1   
kode2                      : ok=58   changed=32   unreachable=0    failed=1   
kode3                      : ok=36   changed=6    unreachable=0    failed=0   
kode4                      : ok=5    changed=0    unreachable=0    failed=0   
kode5                      : ok=5    changed=0    unreachable=0    failed=0

Command failed ansible-playbook -i ../ansible/inventory/bgi-openstack -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla  -e horizon_tag=4.0.0.2 -e action=deploy /opt/bgi-kolla/kolla-ansible/ansible/site.yml

there are the following output on kode2 node 
170318  7:50:27 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out)
170318  7:50:27 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1380: Failed to open channel 'openstack' at 'gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567': -110 (Connection timed out)
170318  7:50:27 [ERROR] WSREP: gcs connect failed: Connection timed out
170318  7:50:27 [ERROR] WSREP: wsrep::connect(gcomm://192.168.102.20:4567,192.168.102.21:4567,192.168.102.22:4567) failed: 7
170318  7:50:27 [ERROR] Aborting

170318  7:50:27 [Note] WSREP: Service disconnected.
170318  7:50:28 [Note] WSREP: Some threads may fail to exit.
170318  7:50:28 [Note] /usr/sbin/mysqld: Shutdown complete

so I check the iptable rules and found the clue, the rule on kode1 blocked port 4567.

I collected the output of "iptables-save > iptable.${HOSTNAME}", you can check the attachement.

Revision history for this message

MarginHu (margin2017) wrote on 2017-03-18:

#13

iptables on 3 controller node. Edit (3.5 KiB, application/zip)

Revision history for this message

Serge Radinovich (srad015) wrote on 2017-11-25:

#14

I wiped /etc/ansible and re-installed kolla-ansible to fix this issue.

kolla

destroy tool is incomplete

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches