MariaDB - InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch

[James Reno]

Ubuntu-Release: (ubuntu-jammy, 22.04, x86_64/amd64; Kernel: 5.15.0-60-generic)
Affected Package: mariadb-server-10.6 = 10.6.12-0ubuntu0.22.04.1

Description:
It looks like the most recent update to mariadb-server-10.6 (https://bugs.launchpad.net/ubuntu/+source/mariadb-10.3/+bug/2006882) may have introduced a regression causing lockups as a result of work completed under one of the following:

[MDEV-24911] Missing warning before [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.mutex - Jira

[MDEV-24258] Merge dict_sys.mutex into dict_sys.latch - Jira

[MDEV-26827] Make page flushing even faster - Jira

We have a large zabbix installation with >398GB history_uint table and ever since this upgrade the mariadb server has been locking up within 2-12 hours with the following error:

[ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch

Downgrade to previous package version 10.6.11 or 10.6.7 resolves the lockup and our platform remains stable.

I have not been able to get a proper crash-dump as the mariadb-server does not crash, it simply hangs -- and sometimes the watchdog auto-restarts it (though not for several hours).

[Otto Kekäläinen (otto)]

Seems this is an upstream issue (=not caused by the packaging in Debian or Ubuntu). Can you please report it to jira.mariadb.org? Thanks!

[James Reno (jameno123)]

Working on the jira report for mariadb however wanted to make sure the ubutu team was aware that a recent patch is causing a major regression / breakage of work.

Would you be able to push the older 10.6.11 packages back to the repo so they are available as an installation option with apt-policy?

[James Reno (jameno123)]

Re: previous comment:

We would like to revert back to the functioning version but the package was removed from the repository which forced us to downgrade all the way back to 10.6.7 (which has its own issues). We would like to be able to restore services back the functioning/previous version which was 10.6.11 but we cant because we've not been able to find them anywhere in any of the repos.

[Faustin (fauust)]

Hi James!
I am not sure what will happen with 10.6.11 and why they are away from Ubuntu repositories (on Debian you could have used snapshots but I am not sure that Ubuntu propose it).

I see 2 options for you to rollback to 10.6.11:
- build the 10.6.11 version. It's not that difficult, see https://mariadb.com/kb/en/building-mariadb-on-debian/;
- use MariaDB Foundation packages from https://archive.mariadb.org/mariadb-10.6.11/repo/ubuntu/.

Hope that helps before a fix is provided.
Faustin

[Otto Kekäläinen (otto)]

All versions of MariaDB in Debian and Ubuntu are available publicly in Debian and Ubuntu archives and build systems. You can download 10.6.11 in Ubuntu e.g. from https://launchpad.net/ubuntu/+source/mariadb-10.6/1:10.6.11-0ubuntu0.22.04.1 -> https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa/+build/24860437

[Otto Kekäläinen (otto)]

Upstream Jira https://jira.mariadb.org/browse/MDEV-24911 seems to still be open, so this is not fixed by latest release MariaDB Server 10.6.16.