rsync issues with SST/IST on joiner

[Raghavendra D Prabhu]

Following issues with SST and IST on joiner with rsync SST:

a) When SST fails, the rsync daemon remains.

b) When SST is tried again, the rsync fails with pid error.

In step a), if there is a failure the daemon must be killed OR it
shouldn't fail in b) with that error.

During IST, it fails as:

        group UUID = 11264cec-06e6-11e2-0800-61616b1fc754
130417 0:00:36 [Note] WSREP: Flow-control interval: [12, 23]
130417 0:00:36 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 17)
130417 0:00:36 [Note] WSREP: State transfer required:
        Group state: 11264cec-06e6-11e2-0800-61616b1fc754:17
        Local state: 11264cec-06e6-11e2-0800-61616b1fc754:3
130417 0:00:36 [Note] WSREP: New cluster view: global state: 11264cec-06e6-11e2-0800-61616b1fc754:17, view# 6: Primary, number of nodes: 2, my index: 1, protocol version 2
130417 0:00:36 [Warning] WSREP: Gap in state sequence. Need state transfer.
130417 0:00:38 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '10.0.2.154' --auth 'root:test' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2656''
/usr//bin/wsrep_sst_common: line 94: /dev/stderr: Permission denied
130417 0:00:38 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_rsync --role 'joiner' --address '10.0.2.154' --auth 'root:test' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2656'
        Read: '(null)'
130417 0:00:38 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '10.0.2.154' --auth 'root:test' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '2656': 1 (Operation not permitted)
130417 0:00:38 [ERROR] WSREP: Failed to prepare for 'rsync' SST. Unrecoverable.
130417 0:00:38 [ERROR] Aborting

[Raghavendra D Prabhu (raghavendra-prabhu)]

Similar issue highlighted in lp:1143052 is where rsync daemon is running between ISTs and causes issues.

[Raghavendra D Prabhu (raghavendra-prabhu)]

With 5.5.30 I don't see any issues since there is cleanup in place, however, if rsync daemon is already running, subsequent SST and IST can fail. This means rsync daemon has to be killed manually since the fact it is running is abnormal because the script kills it when quitting.

[Raghavendra D Prabhu (raghavendra-prabhu)]

One thing, which I see is in wsrep_rsync_sst -- trap cleanup_joiner EXIT -- the cleanup_joiner is not called when TERM etc. are sent, so will require manual cleanup if the parent script is killed with those signals.

[Mrten (bugzilla-ii)]

I regularly have a rsync daemon left running, usually when the upgrade fails for some reason.

I normally trap EXIT SIGTERM SIGKILL SIGINT. Any reason not to add those? I suppose a crash is not exit...

[Alex Yurchenko (ayurchen)]

The full code actually is

    trap "exit 32" HUP PIPE
    trap "exit 3" INT TERM
    trap cleanup_joiner EXIT

So SIGHUP, SIGPIPE, SIGINT and SIGTERM are converted into exit, and cleanup_joiner is supposed to run on exit. IIRC SIGKILL cannot be trapped, it's -9.

What comes to me is that perhaps SIGABRT may be inherited from parent and must be trapped as well?

[Raghavendra D Prabhu (raghavendra-prabhu)]

The fix released in this case is for xtrabackup - to cleanup_joiner on exit. For killings, manual cleanup needs to be done.

[Alex Yurchenko (ayurchen)]

merged into codership-mysql some time in the past (around r3838 in 5.5 and r3917 in 5.6)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-957