Bug #1229632 “Behavior of pc.ignore_sb in Galera 3” : Bugs : Galera

Alex Yurchenko (ayurchen) on 2013-09-24

Changed in galera:
assignee:	nobody → Teemu Ollakka (teemu-ollakka)
milestone:	none → 3.1

Revision history for this message

Teemu Ollakka (teemu-ollakka) wrote on 2013-09-24:

#1

Could not reproduce this:

Command to start mysqld:

/run/shm/galera/local1/mysql/sbin/mysqld --defaults-file=/run/shm/galera/local1/mysql/etc/my.cnf --user=teemu --basedir=/run/shm/galera/local1/mysql --datadir=/run/shm/galera/local1/mysql/var --plugin-dir=lib/mysql/plugin --pid-file=/run/shm/galera/local1/mysql/var/mysqld.pid --port=3311 --socket=/run/shm/galera/local1/mysql/var/mysqld.sock --skip-external-locking --log_error=/run/shm/galera/local1/mysql/var/hp.err --default-storage-engine=InnoDB --wsrep_provider=/run/shm/galera/local1/galera/lib/libgalera_smm.so --wsrep_cluster_address=gcomm://?pc.ignore_sb=1&gmcast.listen_addr=tcp://gw:10011 --wsrep_start_position=9ac15fea-2513-11e3-bcd4-33878a9222ca:0

Log shows:
2013-09-24 15:21:41 8573 [Note] WSREP: (de6aaaa2-2513-11e3-b799-1ef476e847f8, 'tcp://192.168.3.254:10011') listening at tcp://192.168.3.254:10011

$ netstat -natp |grep 10011
tcp 0 0 192.168.3.254:10011 0.0.0.0:* LISTEN 8573/mysqld

Joining another node to cluster goes as usual.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-09-24:

#2

@Teemu,

I had provided it as

--mysqld=--wsrep_cluster_address=gcomm://?pc.ignore_sb=true --mysqld=--wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:8493"

I couldn't get it listening on that address.

Full commandline:

  perl mysql-test-run.pl \
    --start-and-exit \
    --port-base=7000 \
    --nowarnings \
    --vardir=$node1 \
    --mysqld=--innodb_file_per_table \
    --mysqld=--binlog-format=ROW \
    --mysqld=--wsrep-slave-threads=2 \
    --mysqld=--innodb_autoinc_lock_mode=2 \
    --mysqld=--innodb_locks_unsafe_for_binlog=1 \
    --mysqld=--wsrep-provider=/pxc56/lib/libgalera_smm.so \
    --mysqld=--wsrep_cluster_address=gcomm://?pc.ignore_sb=true \
    --mysqld=--wsrep_sst_receive_address=127.0.0.1:37193 \
    --mysqld=--wsrep_node_incoming_address=127.0.0.1 \
    --mysqld=--wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:44444" \
    --mysqld=--wsrep_sst_method=rsync \
    --mysqld=--wsrep_sst_auth=root: \
    --mysqld=--wsrep_node_address=127.0.0.1 \
    --mysqld=--innodb_flush_method=O_DIRECT \
    --mysqld=--core-file \
    --mysqld=--loose-new \
    --mysqld=--sql-mode=no_engine_substitution \
    --mysqld=--loose-innodb \
    --mysqld=--secure-file-priv= \
    --mysqld=--loose-innodb-status-file=1 \
    --mysqld=--skip-name-resolve \
    --mysqld=--socket=/tmp/socket.sock \
    --mysqld=--log-output=none \
    1st

This didn't listen on 127.0.0.1:44444

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-09-24:

#3

As it works when provided differently, the bug may be with how the address is provided.

Revision history for this message

Teemu Ollakka (teemu-ollakka) wrote on 2013-09-25:

#4

I still cannot reproduce this. Could you start the process and post the mysqld command with arguments as it is seen in processlist.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-09-26:

#5

@Teemu,

Ack, I will test it again and post it here.

Revision history for this message

Raghavendra D Prabhu (raghavendra-prabhu) wrote on 2013-09-26:

#6

Download full text (11.1 KiB)

/tmp/run.sh

defined(@array) is deprecated at lib/mtr_cases.pm line 339.
(Maybe you should just omit the defined()?)
defined(@array) is deprecated at mysql-test-run.pl line 497.
(Maybe you should just omit the defined()?)
Logging: mysql-test-run.pl --start-and-exit --port-base=7000 --nowarnings --vardir=/tmp/node1 --mysqld=--innodb_file_per_table --mysqld=--binlog-format=ROW --mysqld=--wsrep-slave-threads=2 --mysqld=--innodb_autoinc_lock_mode=2 --mysqld=--innodb_locks_unsafe_for_binlog=1 --mysqld=--wsrep-provider=/pxc56/lib/libgalera_smm.so --mysqld=--wsrep_sst_receive_address=127.0.0.1:37193 --mysqld=--wsrep_node_incoming_address=127.0.0.1 --mysqld=--wsrep_provider_options=gmcast.listen_addr=tcp://127.0.0.1:44444 --mysqld=--wsrep_sst_method=rsync --mysqld=--wsrep_sst_auth=root: --mysqld=--wsrep_node_address=127.0.0.1 --mysqld=--innodb_flush_method=O_DIRECT --mysqld=--core-file --mysqld=--loose-new --mysqld=--sql-mode=no_engine_substitution --mysqld=--loose-innodb --mysqld=--secure-file-priv= --mysqld=--loose-innodb-status-file=1 --mysqld=--skip-name-resolve --mysqld=--socket=/tmp/socket.sock --mysqld=--log-output=none 1st
2013-09-26 17:21:43 28293 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)

2013-09-26 17:21:43 28293 [Warning] Buffered warning: Changed limits: table_cache: 431 (requested 2000)

2013-09-26 17:21:43 28293 [Note] Binlog end
2013-09-26 17:21:43 28293 [Note] Shutting down plugin 'CSV'
2013-09-26 17:21:43 28293 [Note] Shutting down plugin 'MyISAM'
MySQL Version 5.6.13
Using binlog format 'ROW'
Checking supported features...
- SSL connections supported
- binaries are debug compiled
Collecting tests...
Removing old var directory...
Creating var directory '/tmp/node1'...
Installing system database...
Using server port 41683

==============================================================================

TEST RESULT TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD -300, with reserved ports 7000..7009
worker[1]
Started [mysqld.1 - pid: 28329, winpid: 28329]
worker[1] Using config for test main.1st
worker[1] Port and socket path for server(s):
worker[1] mysqld.1 7001 /tmp/node1/tmp/mysqld.1.sock
worker[1] Server(s) started, not waiting for them to finish

>>p mysqld
raghave+ 28329 1.7 1.4 696276 115860 pts/18 Sl 17:21 0:00 /media/Vone/data/pxc56/bin/mysqld --defaults-group-suffix=.1 --defaults-file=/tmp/node1/my.cnf --log-output=file --loose-debug-sync-timeout=600 --default-storage-engine=MyISAM --default-tmp-storage-engine=MyISAM --loose-skip-log-bin --innodb_file_per_table --wsrep-slave-threads=2 --innodb_autoinc_lock_mode=2 --innodb_locks_unsafe_for_binlog=1 --wsrep-provider=/pxc56/lib/libgalera_smm.so --wsrep_sst_receive_address=127.0.0.1:37193 --wsrep_node_incoming_address=127.0.0.1 --wsrep_provider_options=gmcast.listen_addr=tcp://127.0.0.1:44444 --wsrep_sst_method=rsync --wsrep_sst_auth=root: --wsrep_node_address=127.0.0.1 --innodb_flush_method=O_DIRECT --core-file --loose-new --sql-mode=no_engine_substitution --loose-innodb --secure-file-priv= -...

/tmp/run.sh

defined(@array) is deprecated at lib/mtr_cases.pm line 339.
        (Maybe you should just omit the defined()?)
defined(@array) is deprecated at mysql-test-run.pl line 497.
        (Maybe you should just omit the defined()?)
Logging: mysql-test-run.pl  --start-and-exit --port-base=7000 --nowarnings --vardir=/tmp/node1 --mysqld=--innodb_file_per_table --mysqld=--binlog-format=ROW --mysqld=--wsrep-slave-threads=2 --mysqld=--innodb_autoinc_lock_mode=2 --mysqld=--innodb_locks_unsafe_for_binlog=1 --mysqld=--wsrep-provider=/pxc56/lib/libgalera_smm.so --mysqld=--wsrep_sst_receive_address=127.0.0.1:37193 --mysqld=--wsrep_node_incoming_address=127.0.0.1 --mysqld=--wsrep_provider_options=gmcast.listen_addr=tcp://127.0.0.1:44444 --mysqld=--wsrep_sst_method=rsync --mysqld=--wsrep_sst_auth=root: --mysqld=--wsrep_node_address=127.0.0.1 --mysqld=--innodb_flush_method=O_DIRECT --mysqld=--core-file --mysqld=--loose-new --mysqld=--sql-mode=no_engine_substitution --mysqld=--loose-innodb --mysqld=--secure-file-priv= --mysqld=--loose-innodb-status-file=1 --mysqld=--skip-name-resolve --mysqld=--socket=/tmp/socket.sock --mysqld=--log-output=none 1st
2013-09-26 17:21:43 28293 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)

2013-09-26 17:21:43 28293 [Warning] Buffered warning: Changed limits: table_cache: 431 (requested 2000)

2013-09-26 17:21:43 28293 [Note] Binlog end
2013-09-26 17:21:43 28293 [Note] Shutting down plugin 'CSV'
2013-09-26 17:21:43 28293 [Note] Shutting down plugin 'MyISAM'
MySQL Version 5.6.13
Using binlog format 'ROW'
Checking supported features...
 - SSL connections supported
 - binaries are debug compiled
Collecting tests...
Removing old var directory...
Creating var directory '/tmp/node1'...
Installing system database...
Using server port 41683

==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD -300, with reserved ports 7000..7009
worker[1]
Started [mysqld.1 - pid: 28329, winpid: 28329]
worker[1] Using config for test main.1st
worker[1] Port and socket path for server(s):
worker[1] mysqld.1  7001  /tmp/node1/tmp/mysqld.1.sock
worker[1] Server(s) started, not waiting for them to finish

>>p mysqld
raghave+ 28329  1.7  1.4 696276 115860 pts/18  Sl   17:21   0:00 /media/Vone/data/pxc56/bin/mysqld --defaults-group-suffix=.1 --defaults-file=/tmp/node1/my.cnf --log-output=file --loose-debug-sync-timeout=600 --default-storage-engine=MyISAM --default-tmp-storage-engine=MyISAM --loose-skip-log-bin --innodb_file_per_table --wsrep-slave-threads=2 --innodb_autoinc_lock_mode=2 --innodb_locks_unsafe_for_binlog=1 --wsrep-provider=/pxc56/lib/libgalera_smm.so --wsrep_sst_receive_address=127.0.0.1:37193 --wsrep_node_incoming_address=127.0.0.1 --wsrep_provider_options=gmcast.listen_addr=tcp://127.0.0.1:44444 --wsrep_sst_method=rsync --wsrep_sst_auth=root: --wsrep_node_address=127.0.0.1 --innodb_flush_method=O_DIRECT --core-file --loose-new --sql-mode=no_engine_substitution --loose-innodb --secure-file-priv= --loose-innodb-status-file=1 --skip-name-resolve --socket=/tmp/socket.sock --log-output=none --core-file
raghave+ 28427  0.0  0.0  10936  1076 pts/18   S+   17:22   0:00 grep mysqld

>>sudo netstat -antp | =grep LISTEN | grep mysqld
tcp        0      0 0.0.0.0:7001            0.0.0.0:*               LISTEN      28329/mysqld

=========================================================================================================

cat /tmp/run.sh

#!/bin/zsh

cd /media/Vone/data/pxc56/mysql-test

perl mysql-test-run.pl \
    --start-and-exit \
    --port-base=7000 \
    --nowarnings \
    --vardir=/tmp/node1 \
    --mysqld=--innodb_file_per_table \
    --mysqld=--binlog-format=ROW \
    --mysqld=--wsrep-slave-threads=2 \
    --mysqld=--innodb_autoinc_lock_mode=2 \
    --mysqld=--innodb_locks_unsafe_for_binlog=1 \
    --mysqld=--wsrep-provider=/pxc56/lib/libgalera_smm.so \
    --mysqld=--wsrep_cluster_address=gcomm://?pc.ignore_sb=true \
    --mysqld=--wsrep_sst_receive_address=127.0.0.1:37193 \
    --mysqld=--wsrep_node_incoming_address=127.0.0.1 \
    --mysqld=--wsrep_provider_options="gmcast.listen_addr=tcp://127.0.0.1:44444" \
    --mysqld=--wsrep_sst_method=rsync \
    --mysqld=--wsrep_sst_auth=root: \
    --mysqld=--wsrep_node_address=127.0.0.1 \
    --mysqld=--innodb_flush_method=O_DIRECT \
    --mysqld=--core-file \
    --mysqld=--loose-new \
    --mysqld=--sql-mode=no_engine_substitution \
    --mysqld=--loose-innodb \
    --mysqld=--secure-file-priv= \
    --mysqld=--loose-innodb-status-file=1 \
    --mysqld=--skip-name-resolve \
    --mysqld=--socket=/tmp/socket.sock \
    --mysqld=--log-output=none \
    1st

=================================================================================================

As you can see it is not listening on 44444,

Error  Log:

CURRENT_TEST: main.1st
2013-09-26 14:51:47 0 [Warning] WSREP: wsrep_sst_receive_address is set to '127.0.0.1:37193' which makes it impossible for another host to reach this one. Please set it to the address which this node can be connected at by other cluster members.
2013-09-26 14:51:47 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-09-26 14:51:47 28329 [Warning] Buffered warning: Could not increase number of max_open_files to more than 1024 (request: 4162)

2013-09-26 14:51:47 28329 [Warning] Buffered warning: Changed limits: table_cache: 431 (requested 2000)

2013-09-26 14:51:47 28329 [Note] WSREP: Read nil XID from storage engines, skipping position init
2013-09-26 14:51:47 28329 [Note] WSREP: wsrep_load(): loading provider library '/pxc56/lib/libgalera_smm.so'
2013-09-26 14:51:47 28329 [Note] WSREP: wsrep_load(): Galera 3.0(r155) by Codership Oy <info@codership.com> loaded successfully.
2013-09-26 14:51:47 28329 [Note] WSREP: CRC-32C: using hardware acceleration.
2013-09-26 14:51:47 28329 [Warning] WSREP: Could not open saved state file for reading: /tmp/node1/mysqld.1/data//grastate.dat
2013-09-26 14:51:47 28329 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2013-09-26 14:51:47 28329 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; gcache.dir = /tmp/node1/mysqld.1/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /tmp/node1/mysqld.1/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; gmcast.listen_addr = tcp://127.0.0.1:44444; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.proto_max = 5
2013-09-26 14:51:47 28329 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2013-09-26 14:51:47 7f31e9cbb740 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
2013-09-26 14:51:47 7f31e9cbb740 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
2013-09-26 14:51:47 7f31e9cbb740 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2013-09-26 14:51:47 28329 [Note] InnoDB: The InnoDB memory heap is disabled
2013-09-26 14:51:47 28329 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-09-26 14:51:47 28329 [Note] InnoDB: Compressed tables use zlib 1.2.8
2013-09-26 14:51:47 28329 [Note] InnoDB: Using Linux native AIO
2013-09-26 14:51:47 28329 [Note] InnoDB: Not using CPU crc32 instructions
2013-09-26 14:51:47 28329 [Note] InnoDB: Initializing buffer pool, size = 8.0M
2013-09-26 14:51:47 28329 [Note] InnoDB: Completed initialization of buffer pool
2013-09-26 14:51:47 28329 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: OPEN: Invalid argument, continuing anyway
2013-09-26 14:51:47 28329 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662
2013-09-26 14:51:47 28329 [ERROR] InnoDB: Failed to set O_DIRECT on file ./ibdata1: OPEN: Invalid argument, continuing anyway
2013-09-26 14:51:47 28329 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662
2013-09-26 14:51:47 28329 [Note] InnoDB: Highest supported file format is Barracuda.
2013-09-26 14:51:47 28329 [Note] InnoDB: 128 rollback segment(s) are active.
2013-09-26 14:51:47 28329 [Note] InnoDB: Waiting for purge to start
2013-09-26 14:51:47 28329 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.13-rc60.4 started; log sequence number 1625977
2013-09-26 14:51:47 28329 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 05b4f95c-26a2-11e3-bec6-f04da257636f.
2013-09-26 14:51:47 28329 [Note] RSA private key file not found: /tmp/node1/mysqld.1/data//private_key.pem. Some authentication plugins will not work.
2013-09-26 14:51:47 28329 [Note] RSA public key file not found: /tmp/node1/mysqld.1/data//public_key.pem. Some authentication plugins will not work.
2013-09-26 14:51:47 28329 [Note] Server hostname (bind-address): '*'; port: 7001
2013-09-26 14:51:47 28329 [Note] IPv6 is not available.
2013-09-26 14:51:47 28329 [Note]   - '0.0.0.0' resolves to '0.0.0.0';
2013-09-26 14:51:47 28329 [Note] Server socket created on IP: '0.0.0.0'.
2013-09-26 14:51:47 28329 [Warning] 'user' entry 'root@Archie' ignored in --skip-name-resolve mode.
2013-09-26 14:51:47 28329 [Warning] 'proxies_priv' entry '@ root@Archie' ignored in --skip-name-resolve mode.
2013-09-26 14:51:47 28329 [ERROR] InnoDB: Failed to set O_DIRECT on file ./mysql/slave_master_info.ibd: OPEN: Invalid argument, continuing anyway
2013-09-26 14:51:47 28329 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662
2013-09-26 14:51:47 28329 [ERROR] InnoDB: Failed to set O_DIRECT on file ./mysql/slave_worker_info.ibd: OPEN: Invalid argument, continuing anyway
2013-09-26 14:51:47 28329 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662
2013-09-26 14:51:47 28329 [ERROR] InnoDB: Failed to set O_DIRECT on file ./mysql/slave_relay_log_info.ibd: OPEN: Invalid argument, continuing anyway
2013-09-26 14:51:47 28329 [ERROR] InnoDB: O_DIRECT is known to result in 'Invalid argument' on Linux on tmpfs, see MySQL Bug#26662
2013-09-26 14:51:47 28329 [Note] Event Scheduler: Loaded 0 events
2013-09-26 14:51:47 28329 [Note] /media/Vone/data/pxc56/bin/mysqld: ready for connections.
Version: '5.6.13-24rc2.7.3-debug-log'  socket: '/tmp/socket.sock'  port: 7001  Percona XtraDB Cluster (GPL) 5.6.12-24rc2.7.3, Revision 427, wsrep_24.0.r442

Revision history for this message

Teemu Ollakka (teemu-ollakka) wrote on 2013-09-26:

#7

It looks like --wsrep_cluster_address param is not passed to mysqld process at all and therefore galera wsrep provider does not start.

Revision history for this message

Alex Yurchenko (ayurchen) wrote on 2014-03-27:

#8

can't seem to reproduce

Affects		Status	Importance	Assigned to	Milestone
	Galera	Status tracked in 3.x
	3.x	Invalid	Undecided	Teemu Ollakka	Galera 25.3.5

Galera

Behavior of pc.ignore_sb in Galera 3

Bug Description

Other bug subscribers

Remote bug watches