Database backup may fail if using a different image than running mariadb

[Mark Goddard]

# Steps to reproduce

* Deploy MariaDB using one mariadb-server container image
* Build or pull another mariadb-server image, containing a more recent version of mariadb
* Run kolla-ansible mariadb_backup

# Expected results

Backup completes successfully

# Actual results

Backup may fail with a segmentation fault.

For example:

    [00] 2024-02-07 12:15:19 Finished backing up non-InnoDB tables and files
    240207 12:15:19 [ERROR] mysqld got signal 11 ;
    This could be because you hit a bug. It is also possible that this binary
    or one of the libraries it was linked against is corrupt, improperly built,
    or misconfigured. This error can also be caused by malfunctioning hardware.

    To report this bug, see https://mariadb.com/kb/en/reporting-bugs

    We will try our best to scrape up some info that will hopefully help
    diagnose the problem, but since we have already crashed,
    something is definitely wrong and this may fail.

    Server version: 10.6.15-MariaDB-1:10.6.15+maria~ubu2204 source revision: 0d16eb35bc981023ce2f4912e8ecde68ca381f4e
    key_buffer_size=0
    read_buffer_size=131072
    max_used_connections=0
    max_threads=1
    thread_count=0
    It is possible that mysqld could use up to
    key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6161 K bytes of memory
    Hope that's ok; if not, decrease some variables in the equation.

    Thread pointer: 0x0
    Attempting backtrace. You can use the following information to find out
    where mysqld died. If you see no messages after this, something went
    terribly wrong...
    stack_bottom = 0x0 thread_stack 0x49000
    Printing to addr2line failed
    mariabackup(my_print_stacktrace+0x32)[0x56529225a8d2]
    mariabackup(handle_fatal_signal+0x488)[0x565291d88cc8]
    /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f10502c3520]
    /lib/x86_64-linux-gnu/libc.so.6(opendir+0x22)[0x7f1050367312]
    mariabackup(+0x713154)[0x5652919d5154]
    mariabackup(_Z12backup_startP7ds_ctxtS0_R14CorruptedPages+0x10b)[0x5652919d87cb]
    mariabackup(+0x6ff59f)[0x5652919c159f]
    mariabackup(main+0x182)[0x565291965d42]
    /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f10502aad90]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f10502aae40]
    mariabackup(_start+0x25)[0x5652919ac055]
    The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
    information that should help you find out what is causing the crash.
    Writing a core file...
    Working directory at /var/lib/mysql
    Resource Limits:
    Limit Soft Limit Hard Limit Units
    Max cpu time unlimited unlimited seconds
    Max file size unlimited unlimited bytes
    Max data size unlimited unlimited bytes
    Max stack size 8388608 unlimited bytes
    Max core file size unlimited unlimited bytes
    Max resident set unlimited unlimited bytes
    Max processes unlimited unlimited processes
    Max open files 1048576 1048576 files
    Max locked memory 65536 65536 bytes
    Max address space unlimited unlimited bytes
    Max file locks unlimited unlimited locks
    Max pending signals 4076729 4076729 signals
    Max msgqueue size 819200 819200 bytes
    Max nice priority 0 0
    Max realtime priority 0 0
    Max realtime timeout unlimited unlimited us
    Core pattern: |/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E

    Kernel version: Linux version 5.15.0-87-generic (buildd@lcy02-amd64-011) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023

    /usr/local/bin/kolla_mariadb_backup.sh: line 7: 11 Segmentation fault (core dumped) mariabackup --defaults-file=/etc/mysql/my.cnf --backup --stream=xbstream --history=$LAST_FULL_DATE
            12 Done | gzip > $BACKUP_DIR/mysqlbackup-$(date +%d-%m-%Y-%s).qp.xbc.xbs.gz
  stderr_lines: <omitted>
  stdout: |-
    Taking a full backup
  stdout_lines: <omitted>

[Mark Goddard (mgoddard)]

One option to avoid this could be to query the image that the MariaDB servers are running, and use the same one to perform the backup.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/913901