kolla-ansible

kolla-ansible 13.0.0.0rc1 maria-db won't get live

Bug #1947534 reported by Vahid Alimohamadi on 2021-10-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Fix Released	Medium	Unassigned

Bug Description

TASK [mariadb : Check MariaDB service port liveness] ************************************************
fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 10, "msg": "Timeout when waiting for search string MariaDB in 192.168.54.12:3306"}

then stuck at
RUNNING HANDLER [mariadb : Wait for first MariaDB service port liveness]
and count down to zero and failure.

I tried to access mariadb using mysql client and got timeout but there's no error log from container:

mariadb image:
quay.io/openstack.kolla/ubuntu-source-mariadb-server xena eb5e1822ecce 18 hours ago 624MB

mariadb container log output:

...
INFO:__main__:Setting permission for /var/lib/mysql/mysql/time_zone.MAI
INFO:__main__:Setting permission for /var/lib/mysql/mysql/index_stats.MAD
INFO:__main__:Setting permission for /var/lib/mysql/mysql/procs_priv.MAI
INFO:__main__:Setting permission for /var/lib/mysql/mysql/servers.frm
INFO:__main__:Setting permission for /var/lib/mysql/performance_schema/db.opt
++ cat /run_command
+ CMD=/usr/bin/mysqld_safe
+ ARGS=
+ sudo kolla_copy_cacerts
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ : /var/log/kolla/mariadb
++ [[ ! -d /var/log/kolla/mariadb ]]
+++ stat -c %a /var/log/kolla/mariadb
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/mariadb
++ [[ -n '' ]]
++ [[ -n '' ]]
++ [[ -n 0 ]]
++ ARGS=--wsrep-new-cluster
+ echo 'Running command: '\''/usr/bin/mysqld_safe --wsrep-new-cluster'\'''
+ exec /usr/bin/mysqld_safe --wsrep-new-cluster
Running command: '/usr/bin/mysqld_safe --wsrep-new-cluster'
211018 00:30:19 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'.
211018 00:30:19 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/

OS:
I tried on CentOS 8 stream, ubuntu 20.04 LTS and ubuntu 21.04 LTS HP G7 bare-metal.
Kernel: Linux os 5.11.0-37-generic #41-Ubuntu SMP Mon Sep 20 16:39:20 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
kolla-ansible version: 13.0.0.0rc1 all-in-one
Docker version:20.10.9, build c2ea9bc

kolla global.conf:

config_strategy: "COPY_ALWAYS"
kolla_base_distro: "ubuntu"
kolla_install_type: "source"
openstack_release: "xena"
kolla_internal_vip_address: "192.168.54.12"
network_interface: eno1
neutron_external_interface: eno2
neutron_plugin_agent: "openvswitch"
enable_haproxy: "no"
enable_ceilometer: "yes"
enable_cinder: "yes"
enable_cinder_backup: "no"
enable_cinder_backend_lvm: "no"
enable_cloudkitty: "yes"
enable_gnocchi: "yes"
enable_neutron_provider_networks: "yes"
ceph_cinder_keyring: "ceph.client.admin.keyring"
ceph_cinder_user: "admin"
ceph_cinder_pool_name: "volumes"
fernet_token_expiry: 86400
cinder_backend_ceph: "yes"
cinder_volume_group: "volumes"
nova_compute_virt_type: "kvm"
nova_console: "novnc"

This config works on kolla 12.2.0 and wallaby release but those images are on dockerhub not quay.io

See original description

Tags:

Vahid Alimohamadi (cybercoder) on 2021-10-18

tags:	added: all-in-one
tags:	added: mult-node

Vahid Alimohamadi (cybercoder) on 2021-10-18

description:	updated
description:	updated

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2021-10-18:

What's in mariadb logs?

Changed in kolla-ansible:
status:	New → Incomplete

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2021-10-18:

The path is in the error report, just need to include its contents: /var/log/kolla/mariadb/mariadb.log

Revision history for this message

Vahid Alimohamadi (cybercoder) wrote on 2021-10-18:

Download full text (12.6 KiB)

mariadb log:

2021-10-18 9:56:41 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
211018 09:56:42 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/
2021-10-18 9:56:42 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-1:10.5.12+maria~focal-log) starting as process 264 ...
2021-10-18 9:56:42 0 [Note] InnoDB: Uses event mutexes
2021-10-18 9:56:42 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-18 9:56:42 0 [Note] InnoDB: Number of pools: 1
2021-10-18 9:56:42 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-10-18 9:56:42 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-10-18 9:56:42 0 [Note] InnoDB: Using Linux native AIO
2021-10-18 9:56:42 0 [Note] InnoDB: Initializing buffer pool, total size = 8589934592, chunk size = 134217728
2021-10-18 9:56:43 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-18 9:56:43 0 [Note] InnoDB: 128 rollback segments are active.
2021-10-18 9:56:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-10-18 9:56:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-10-18 9:56:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-10-18 9:56:43 0 [Note] InnoDB: 10.5.12 started; log sequence number 45094; transaction id 20
2021-10-18 9:56:43 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-10-18 9:56:43 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-10-18 9:56:43 0 [Note] InnoDB: Buffer pool(s) load completed at 211018 9:56:43
2021-10-18 9:56:43 0 [Note] Reading of all Master_info entries succeeded
2021-10-18 9:56:43 0 [Note] Added new Master_info '' to hash table
2021-10-18 9:56:43 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.5.12-MariaDB-1:10.5.12+maria~focal-log' socket: '/run/mysqld/mysqld.sock' port: 0 mariadb.org binary distribution
2021-10-18 9:56:43 0 [Note] /usr/sbin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
2021-10-18 9:56:43 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-10-18 9:56:43 0 [Note] InnoDB: FTS optimize thread exiting.
2021-10-18 9:56:43 0 [Note] InnoDB: Starting shutdown...
2021-10-18 9:56:43 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-10-18 9:56:43 0 [Note] InnoDB: Buffer pool(s) dump completed at 211018 9:56:43
2021-10-18 9:56:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-10-18 9:56:44 0 [Note] InnoDB: Shutdown completed; log sequence number 45106; transaction id 23
2021-10-18 9:56:44 0 [Note] /usr/sbin/mariadbd: Shutdown complete

mariadb log:

2021-10-18  9:56:41 1 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
211018 09:56:42 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/
2021-10-18  9:56:42 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-1:10.5.12+maria~focal-log) starting as process 264 ...
2021-10-18  9:56:42 0 [Note] InnoDB: Uses event mutexes
2021-10-18  9:56:42 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-18  9:56:42 0 [Note] InnoDB: Number of pools: 1
2021-10-18  9:56:42 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-10-18  9:56:42 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-10-18  9:56:42 0 [Note] InnoDB: Using Linux native AIO
2021-10-18  9:56:42 0 [Note] InnoDB: Initializing buffer pool, total size = 8589934592, chunk size = 134217728
2021-10-18  9:56:43 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-18  9:56:43 0 [Note] InnoDB: 128 rollback segments are active.
2021-10-18  9:56:43 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-10-18  9:56:43 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-10-18  9:56:43 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-10-18  9:56:43 0 [Note] InnoDB: 10.5.12 started; log sequence number 45094; transaction id 20
2021-10-18  9:56:43 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-10-18  9:56:43 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-10-18  9:56:43 0 [Note] InnoDB: Buffer pool(s) load completed at 211018  9:56:43
2021-10-18  9:56:43 0 [Note] Reading of all Master_info entries succeeded
2021-10-18  9:56:43 0 [Note] Added new Master_info '' to hash table
2021-10-18  9:56:43 0 [Note] /usr/sbin/mariadbd: ready for connections.
Version: '10.5.12-MariaDB-1:10.5.12+maria~focal-log'  socket: '/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2021-10-18  9:56:43 0 [Note] /usr/sbin/mariadbd (initiated by: root[root] @ localhost []): Normal shutdown
2021-10-18  9:56:43 0 [Note] Event Scheduler: Purging the queue. 0 events
2021-10-18  9:56:43 0 [Note] InnoDB: FTS optimize thread exiting.
2021-10-18  9:56:43 0 [Note] InnoDB: Starting shutdown...
2021-10-18  9:56:43 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2021-10-18  9:56:43 0 [Note] InnoDB: Buffer pool(s) dump completed at 211018  9:56:43
2021-10-18  9:56:44 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-10-18  9:56:44 0 [Note] InnoDB: Shutdown completed; log sequence number 45106; transaction id 23
2021-10-18  9:56:44 0 [Note] /usr/sbin/mariadbd: Shutdown complete

211018 09:56:44 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended
211018 09:56:47 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql/
211018 09:56:47 mysqld_safe WSREP: Running position recovery with --disable-log-error  --pid-file='/var/lib/mysql//os-recover.pid'
211018 09:56:48 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2021-10-18  9:56:48 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-1:10.5.12+maria~focal-log) starting as process 234 ...
2021-10-18  9:56:48 0 [Note] WSREP: Loading provider /usr/lib/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2021-10-18  9:56:48 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2021-10-18  9:56:48 0 [Note] WSREP: wsrep_load(): Galera 26.4.9(r819f29cb) by Codership Oy <info@codership.com> loaded successfully.
2021-10-18  9:56:48 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-10-18  9:56:48 0 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
2021-10-18  9:56:48 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2021-10-18  9:56:48 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 0
UUID: 00000000-0000-0000-0000-000000000000
Seqno: -1 - -1
Offset: -1
Synced: 0
2021-10-18  9:56:48 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
2021-10-18  9:56:48 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.54.12; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://192.168.54.12:4567; gmcast.segment = 0; gmcast.version
2021-10-18  9:56:48 0 [Note] WSREP: Start replication
2021-10-18  9:56:48 0 [Note] WSREP: Connecting with bootstrap option: 1
2021-10-18  9:56:48 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2021-10-18  9:56:48 0 [Note] WSREP: protonet asio version 0
2021-10-18  9:56:48 0 [Note] WSREP: Using CRC-32C for message checksums.
2021-10-18  9:56:48 0 [Note] WSREP: backend: asio
2021-10-18  9:56:48 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2021-10-18  9:56:48 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2021-10-18  9:56:48 0 [Note] WSREP: restore pc from disk failed
2021-10-18  9:56:48 0 [Note] WSREP: GMCast version 0
2021-10-18  9:56:48 0 [Note] WSREP: (b608aeaa-bb25, 'tcp://192.168.54.12:4567') listening at tcp://192.168.54.12:4567
2021-10-18  9:56:48 0 [Note] WSREP: (b608aeaa-bb25, 'tcp://192.168.54.12:4567') multicast: , ttl: 1
2021-10-18  9:56:48 0 [Note] WSREP: EVS version 1
2021-10-18  9:56:48 0 [Note] WSREP: gcomm: bootstrapping new group 'openstack'
2021-10-18  9:56:48 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery
2021-10-18  9:56:48 0 [Note] WSREP: EVS version upgrade 0 -> 1
2021-10-18  9:56:48 0 [Note] WSREP: PC protocol upgrade 0 -> 1
2021-10-18  9:56:48 0 [Note] WSREP: Node b608aeaa-bb25 state prim
2021-10-18  9:56:48 0 [Note] WSREP: view(view_id(PRIM,b608aeaa-bb25,1) memb {
	b608aeaa-bb25,0
} joined {
} left {
} partitioned {
})
2021-10-18  9:56:48 0 [Note] WSREP: save pc into disk
2021-10-18  9:56:48 0 [Note] WSREP: gcomm: connected
2021-10-18  9:56:48 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2021-10-18  9:56:48 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2021-10-18  9:56:48 0 [Note] WSREP: Opened channel 'openstack'
2021-10-18  9:56:48 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2021-10-18  9:56:48 0 [Note] WSREP: Starting new group from scratch: b6091d00-2ff9-11ec-8edb-6ec5e823c746
2021-10-18  9:56:48 1 [Note] WSREP: Starting rollbacker thread 1
2021-10-18  9:56:48 0 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b609201e-2ff9-11ec-a5a1-deac7d55e38c
2021-10-18  9:56:48 2 [Note] WSREP: Starting applier thread 2
2021-10-18  9:56:48 0 [Note] WSREP: STATE EXCHANGE: sent state msg: b609201e-2ff9-11ec-a5a1-deac7d55e38c
2021-10-18  9:56:48 0 [Note] WSREP: STATE EXCHANGE: got state msg: b609201e-2ff9-11ec-a5a1-deac7d55e38c from 0 (os)
2021-10-18  9:56:48 0 [Note] WSREP: Quorum results:
	version    = 6,
	component  = PRIMARY,
	conf_id    = 0,
	members    = 1/1 (joined/total),
	act_id     = 0,
	last_appl. = 0,
	protocols  = 2/10/4 (gcs/repl/appl),
	vote policy= 0,
	group UUID = b6091d00-2ff9-11ec-8edb-6ec5e823c746
2021-10-18  9:56:48 0 [Note] WSREP: Flow-control interval: [16, 16]
2021-10-18  9:56:48 0 [Note] WSREP: Restored state OPEN -> JOINED (1)
2021-10-18  9:56:48 0 [Note] WSREP: Member 0.0 (os) synced with group.
2021-10-18  9:56:48 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 1)
2021-10-18  9:56:48 2 [Note] WSREP: ####### processing CC 1, local, ordered
2021-10-18  9:56:48 2 [Note] WSREP: Process first view: b6091d00-2ff9-11ec-8edb-6ec5e823c746 my uuid: b608aeaa-2ff9-11ec-bb25-224438446ee7
2021-10-18  9:56:48 2 [Note] WSREP: Server os connected to cluster at position b6091d00-2ff9-11ec-8edb-6ec5e823c746:1 with ID b608aeaa-2ff9-11ec-bb25-224438446ee7
2021-10-18  9:56:48 2 [Note] WSREP: Server status change disconnected -> connected
ERROR 2002 (HY000): Can't connect to MySQL server on '192.168.54.12' (115)
2021-10-18  9:56:48 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status connected: 1 (Operation not permitted)
2021-10-18  9:56:48 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status connected"
2021-10-18  9:56:48 2 [Note] WSREP: ####### My UUID: b608aeaa-2ff9-11ec-bb25-224438446ee7
2021-10-18  9:56:48 2 [Note] WSREP: Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
2021-10-18  9:56:48 0 [Note] WSREP: Service thread queue flushed.
2021-10-18  9:56:48 2 [Note] WSREP: ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: -1
2021-10-18  9:56:48 2 [Note] WSREP: REPL Protocols: 10 (5)
2021-10-18  9:56:48 2 [Note] WSREP: ####### Adjusting cert position: -1 -> 1
2021-10-18  9:56:48 0 [Note] WSREP: Service thread queue flushed.
2021-10-18  9:56:48 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> b6091d00-2ff9-11ec-8edb-6ec5e823c746:0
2021-10-18  9:56:48 2 [Note] WSREP: ================================================
View:
  id: b6091d00-2ff9-11ec-8edb-6ec5e823c746:1
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
	0: b608aeaa-2ff9-11ec-bb25-224438446ee7, os
=================================================
2021-10-18  9:56:48 2 [Note] WSREP: Server status change connected -> joiner
ERROR 2002 (HY000): Can't connect to MySQL server on '192.168.54.12' (115)
2021-10-18  9:56:48 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status joiner: 1 (Operation not permitted)
2021-10-18  9:56:48 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status joiner"
2021-10-18  9:56:48 2 [Note] WSREP: Server status change joiner -> initializing
ERROR 2002 (HY000): Can't connect to MySQL server on '192.168.54.12' (115)
2021-10-18  9:56:48 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status initializing: 1 (Operation not permitted)
2021-10-18  9:56:48 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status initializing"
2021-10-18  9:56:48 0 [Note] InnoDB: Uses event mutexes
2021-10-18  9:56:48 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-10-18  9:56:48 0 [Note] InnoDB: Number of pools: 1
2021-10-18  9:56:48 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-10-18  9:56:48 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-10-18  9:56:49 0 [Note] InnoDB: Using Linux native AIO
2021-10-18  9:56:49 0 [Note] InnoDB: Initializing buffer pool, total size = 8589934592, chunk size = 134217728
2021-10-18  9:56:49 0 [Note] InnoDB: Completed initialization of buffer pool
2021-10-18  9:56:49 0 [Note] InnoDB: 128 rollback segments are active.
2021-10-18  9:56:49 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-10-18  9:56:49 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-10-18  9:56:49 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-10-18  9:56:49 0 [Note] InnoDB: 10.5.12 started; log sequence number 45118; transaction id 20
2021-10-18  9:56:49 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-10-18  9:56:49 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-10-18  9:56:49 0 [Note] InnoDB: Buffer pool(s) load completed at 211018  9:56:49
2021-10-18  9:56:49 0 [Note] Server socket created on IP: '192.168.54.12'.
2021-10-18  9:56:49 0 [Note] WSREP: wsrep_init_schema_and_SR 0x0
2021-10-18  9:56:49 0 [Note] WSREP: Server initialized
2021-10-18  9:56:49 0 [Note] WSREP: Server status change initializing -> initialized

Vahid Alimohamadi (cybercoder) on 2021-10-18

description:

updated

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2021-10-18:

Triaged; as a workaround please add:

enable_mariadb_clustercheck: yes

Changed in kolla-ansible:
status:	Incomplete → Triaged
importance:	Undecided → Medium

OpenStack Infra (hudson-openstack) on 2021-10-18

Changed in kolla-ansible:
status:	Triaged → In Progress

Revision history for this message

Vahid Alimohamadi (cybercoder) wrote on 2021-10-18 (last edit on 2021-10-18):

It works! I'm so sorry for reporting as a bug but the option you mentioned is not there in the sample global.conf

Thanks anyway.

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2021-10-18:

Don't feel sorry, it's a proper bug. :-) Thank you for reporting.

Revision history for this message

Victor Morales (electrocucaracha) wrote on 2021-10-19:

I got it this error as well in All-in-One setup. These are the MariaDB's errors.

$ docker exec mariadb grep ERROR /var/log/kolla/mariadb/mariadb.log
ERROR 2002 (HY000): Can't connect to MySQL server on '10.0.2.204' (115)
2021-10-19 16:39:26 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status connected: 1 (Operation not permitted)
2021-10-19 16:39:26 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status connected"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.0.2.204' (115)
2021-10-19 16:39:26 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status joiner: 1 (Operation not permitted)
2021-10-19 16:39:26 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status joiner"
ERROR 2002 (HY000): Can't connect to MySQL server on '10.0.2.204' (115)
2021-10-19 16:39:26 2 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep-notify.sh --status initializing: 1 (Operation not permitted)
2021-10-19 16:39:26 2 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep-notify.sh --status initializing"

The documentation mentions[2] that All-in-One setups can run without haproxy and keepalived setting enable_haproxy to false but this enables[3] wsrep-notify.sh execution[1]. Is it correct this disabling logic[1]?

[1] https://review.opendev.org/c/openstack/kolla-ansible/+/786182
[2] https://github.com/openstack/kolla-ansible/blob/stable/xena/etc/kolla/globals.yml#L49-L52
[3] https://github.com/openstack/kolla-ansible/blob/stable/xena/ansible/roles/mariadb/defaults/main.yml#L121

Revision history for this message

Radosław Piliszek (yoctozepto) wrote on 2021-10-19:

Hi Victor, the workaround is in comment #4.
The real fix is in https://review.opendev.org/c/openstack/kolla-ansible/+/814393

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-20: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/814393
Committed: https://opendev.org/openstack/kolla-ansible/commit/b6b7401c0d71298ac32b1437c3a309c6957ef5c0
Submitter: "Zuul (22348)"
Branch: master

commit b6b7401c0d71298ac32b1437c3a309c6957ef5c0
Author: Michal Nasiadka <email address hidden>
Date: Mon Oct 18 12:09:03 2021 +0200

mariadb: Remove wsrep-notify.sh

Closes-Bug: #1947534

Change-Id: I08be074c3633cc4fb0a0bc6c9cb8d03eb5226d89

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-20: Fix proposed to kolla-ansible (stable/xena)

#10

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/814667

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-20: Fix proposed to kolla-ansible (stable/wallaby)

#11

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/814668

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-21: Fix merged to kolla-ansible (stable/xena)

#12

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/814667
Committed: https://opendev.org/openstack/kolla-ansible/commit/70087813c71e6442d26185812b32b2447c8202fe
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 70087813c71e6442d26185812b32b2447c8202fe
Author: Michal Nasiadka <email address hidden>
Date: Mon Oct 18 12:09:03 2021 +0200

mariadb: Remove wsrep-notify.sh

Closes-Bug: #1947534

Change-Id: I08be074c3633cc4fb0a0bc6c9cb8d03eb5226d89
(cherry picked from commit b6b7401c0d71298ac32b1437c3a309c6957ef5c0)

tags:

added: in-stable-xena

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-10-27: Fix merged to kolla-ansible (stable/wallaby)

#13

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/814668
Committed: https://opendev.org/openstack/kolla-ansible/commit/1a1fb864385f2b798be6fe8554cbdc0acb96bf7c
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 1a1fb864385f2b798be6fe8554cbdc0acb96bf7c
Author: Michal Nasiadka <email address hidden>
Date: Mon Oct 18 12:09:03 2021 +0200

mariadb: Do not use wsrep-notify.sh on Debian

Closes-Bug: #1947534

Change-Id: I08be074c3633cc4fb0a0bc6c9cb8d03eb5226d89
(cherry picked from commit b6b7401c0d71298ac32b1437c3a309c6957ef5c0)

tags:

added: in-stable-wallaby

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2021-11-05: Fix included in openstack/kolla-ansible 13.0.0.0rc2

#14

This issue was fixed in the openstack/kolla-ansible 13.0.0.0rc2 release candidate.

Revision history for this message

zhuxiaochun (zhuxiaochun1995) wrote on 2021-11-20:

#15

There is an unexpected } in line 11 of mariadb.json.j2 on branch stable/wallaby, please fix it.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-01-03: Fix included in openstack/kolla-ansible 12.3.0

#16

This issue was fixed in the openstack/kolla-ansible 12.3.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-04-07: Fix included in openstack/kolla-ansible 14.0.0.0rc1

#17

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.