featureset001 - mysql fails to start - WSREP: failed to open gcomm backend connection: 131: No address to connect (FATAL)

Bug #1928869 reported by Harald Jensås
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Unassigned

Bug Description

https://logserver.rdoproject.org/21/791721/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/85d9fdd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2021-05-18 20:19:40 | [ERROR]: Container(s) which failed to be created by podman_container module:
2021-05-18 20:19:40 | ['mysql_wait_bundle']
2021-05-18 20:19:40 | [ERROR]: Container(s) which did not finish after 300 minutes:
2021-05-18 20:19:40 | ['mysql_wait_bundle']
2021-05-18 20:19:40 | 2021-05-18 20:19:40.392824 | fa163ea5-9bc4-f6f3-0e49-000000006da8 | FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['mysql_wait_bundle'], check logs in /var/log/containers/stdouts/"}

https://logserver.rdoproject.org/21/791721/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/85d9fdd/logs/overcloud-controller-0/var/log/containers/mysql/mysqld.log.txt.gz

2021-05-18 20:03:07 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2021-05-18 20:03:07 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
2021-05-18 20:03:07 0 [Note] WSREP: wsrep_load(): Galera 3.31(rXXXX) by Codership Oy <email address hidden> loaded successfully.
2021-05-18 20:03:07 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-05-18 20:03:07 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
2021-05-18 20:03:07 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2021-05-18 20:03:07 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = overcloud-controller-0.internalapi.localdomain; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://172.17
2021-05-18 20:03:07 0 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 00000000-0000-0000-0000-000000000000:-1
2021-05-18 20:03:07 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2021-05-18 20:03:07 0 [Note] WSREP: wsrep_sst_grab()
2021-05-18 20:03:07 0 [Note] WSREP: Start replication
2021-05-18 20:03:07 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2021-05-18 20:03:07 0 [Note] WSREP: protonet asio version 0
2021-05-18 20:03:07 0 [Note] WSREP: Using CRC-32C for message checksums.
2021-05-18 20:03:07 0 [Note] WSREP: backend: asio
2021-05-18 20:03:07 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2021-05-18 20:03:07 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2021-05-18 20:03:07 0 [Note] WSREP: restore pc from disk failed
2021-05-18 20:03:07 0 [Note] WSREP: GMCast version 0
2021-05-18 20:03:07 0 [Warning] WSREP: Failed to resolve tcp://overcloud-controller-0.internalapi.localdomain.:4567
2021-05-18 20:03:07 0 [Warning] WSREP: Failed to resolve tcp://overcloud-controller-1.internalapi.localdomain.:4567
2021-05-18 20:03:07 0 [Warning] WSREP: Failed to resolve tcp://overcloud-controller-2.internalapi.localdomain.:4567
2021-05-18 20:03:07 0 [Note] WSREP: (106da374, 'tcp://172.17.0.231:4567') listening at tcp://172.17.0.231:4567
2021-05-18 20:03:07 0 [Note] WSREP: (106da374, 'tcp://172.17.0.231:4567') multicast: , ttl: 1
2021-05-18 20:03:07 0 [Note] WSREP: EVS version 0
2021-05-18 20:03:07 0 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer 'overcloud-controller-0.internalapi.localdomain.:,overcloud-controller-1.internalapi.localdomain.:,overcloud-controller-2.internalapi.localdomain.:'
2021-05-18 20:03:07 0 [ERROR] WSREP: failed to open gcomm backend connection: 131: No address to connect (FATAL)
  at gcomm/src/gmcast.cpp:connect_precheck():311
2021-05-18 20:03:07 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -131 (State not recoverable)
2021-05-18 20:03:07 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1475: Failed to open channel 'galera_cluster' at 'gcomm://overcloud-controller-0.internalapi.localdomain.,overcloud-controller-1.internalapi.localdomain.,overcloud-controller-2.internalapi.localdomain.': -131 (State not recoverable)
2021-05-18 20:03:07 0 [ERROR] WSREP: gcs connect failed: State not recoverable
2021-05-18 20:03:07 0 [ERROR] WSREP: wsrep::connect(gcomm://overcloud-controller-0.internalapi.localdomain.,overcloud-controller-1.internalapi.localdomain.,overcloud-controller-2.internalapi.localdomain.) failed: 7
2021-05-18 20:03:07 0 [ERROR] Aborting

tags: added: ci
Revision history for this message
Amol Kahat (amolkahat) wrote :
tags: added: promotion-blocker
Revision history for this message
Rabi Mishra (rabi) wrote :
Revision history for this message
Harald Jensås (harald-jensas) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (master)
Changed in tripleo:
status: Triaged → In Progress
tags: added: wallaby-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/792136
Committed: https://opendev.org/openstack/tripleo-common/commit/bac04330bbbe045c8d847b08d150c9ab8e139395
Submitter: "Zuul (22348)"
Branch: master

commit bac04330bbbe045c8d847b08d150c9ab8e139395
Author: Harald Jensås <email address hidden>
Date: Wed May 19 14:35:05 2021 +0200

    Strip final dot of the fqdn hostname and dnsname

    Not all services support an FQDN ending with a dot.

    Closes-Bug: #1928869
    Change-Id: I6b1de8815796b76150f4fed97b7e15dc9adcbbfe

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-common (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/tripleo-common/+/792887

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-common (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/tripleo-common/+/792887
Committed: https://opendev.org/openstack/tripleo-common/commit/99ec07c516b8e4a8ed1fe33d2fc3070b8e1dd3d5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 99ec07c516b8e4a8ed1fe33d2fc3070b8e1dd3d5
Author: Harald Jensås <email address hidden>
Date: Wed May 19 14:35:05 2021 +0200

    Strip final dot of the fqdn hostname and dnsname

    Not all services support an FQDN ending with a dot.

    Closes-Bug: #1928869
    Change-Id: I6b1de8815796b76150f4fed97b7e15dc9adcbbfe
    (cherry picked from commit bac04330bbbe045c8d847b08d150c9ab8e139395)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 15.2.1

This issue was fixed in the openstack/tripleo-common 15.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-common 16.1.0

This issue was fixed in the openstack/tripleo-common 16.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.