Neutron migrations to Ussuri fail randomly on Debian

[Radosław Piliszek]

Upgrades to Ussuri fail sporadically on Debian (used to fail on Ubuntu too) due to Neutron migrations failing.

I have found the issue. It's a regression in MariaDB. 10.3.29 is the affected version and it got itself into Debian May 09: https://metadata.ftp-master.debian.org/changelogs//main/m/mariadb-10.3/mariadb-10.3_10.3.29-0+deb10u1_changelog (the upstream release was May 07). We use upstream in Ubuntu and then it started failing too. Then it fixed itself in late June as 10.3.30 was emergency-released Jun 23 and pushed to upstream repos. CentOS 8 uses 10.3.28 which hangs right before the regression. 😂

the MariaDB bug report: https://jira.mariadb.org/browse/MDEV-25672
10.3.30 changelog: https://mariadb.com/kb/en/mariadb-10330-release-notes/

Debian maintainers are aware of the regression fixing release since Jun 25 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990306 but they have not released even now (Sep 21).
This affects Debian Buster as used in Ussuri and Victoria.

Issue variants:

1) early, e.g., CI-805225-1-debian-ebffa0e, CI-805266-1-debian-95ec869
c613d0b82681 -> c3e9d13c4367
c3e9d13c4367 -> 86274d77933e
oslo_db.exception.DBError: (pymysql.err.InternalError) (1054, "Unknown column '`neutron`.`networks_1`.`admin_state_up`' in 'CHECK'")
[SQL: ALTER TABLE networks MODIFY mtu INTEGER NOT NULL DEFAULT '1500']

2) late, e.g., CI-804337-1-debian-aa0c35c, CI-805266-1-debian-dc33623, -274f3f9, and yoctozepto's local
c613d0b82681 -> c3e9d13c4367
...
e4e236b0e1ff -> e88badaa9591
oslo_db.exception.DBError: (pymysql.err.InternalError) (1054, "Unknown column '`neutron`.`subnetpools_1`.`shared`' in 'CHECK'")
[SQL: ALTER TABLE subnetpools ALTER COLUMN shared SET DEFAULT false]

PS: after the investigation, I still wonder how come this issue is random in CI, yet seems permanent in my case (and also, why I'm getting the later variant 2 rather than the early one)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/810161

[Michal Arbet (michalarbet)]

Fixed in http://buster-victoria.debian.net/debian/pool/buster-victoria-backports/main/d/designate/

[Radosław Piliszek (yoctozepto)]

What does it have to do with Designate?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/810161
Committed: https://opendev.org/openstack/kolla/commit/b5422a6bc27797144da1ea973cec24a5cf46d2eb
Submitter: "Zuul (22348)"
Branch: master

commit b5422a6bc27797144da1ea973cec24a5cf46d2eb
Author: Radosław Piliszek <email address hidden>
Date: Tue Sep 21 07:40:58 2021 +0000

    [debian] Use upstream MariaDB

    This preserves the Debian-provided minor version - 10.5 in case
    of Bullseye in Wallaby and Xena - and will be backported as
    a fix to Victoria and Ussuri (where it will follow Buster and its
    10.3).

    Related-Bug: #1944410
    Change-Id: I9704e5371a410cb1f69ec0367174e2b368961dc1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/810744

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla/+/810747

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla/+/810749

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/810749
Committed: https://opendev.org/openstack/kolla/commit/23af39b046a1c1ce9802d4f358410af99d2f10b9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 23af39b046a1c1ce9802d4f358410af99d2f10b9
Author: Radosław Piliszek <email address hidden>
Date: Tue Sep 21 07:40:58 2021 +0000

    [debian] Use upstream MariaDB

    This preserves the Debian-provided minor version - 10.5 in case
    of Bullseye in Wallaby and Xena - and is backported as
    a fix to Victoria and Ussuri (where it follows Buster and its
    10.3).

    Closes-Bug: #1944410
    Change-Id: I9704e5371a410cb1f69ec0367174e2b368961dc1
    (cherry picked from commit b5422a6bc27797144da1ea973cec24a5cf46d2eb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/810747
Committed: https://opendev.org/openstack/kolla/commit/8c9bcb0e7dcb652227995db1af3e9f6205943371
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 8c9bcb0e7dcb652227995db1af3e9f6205943371
Author: Radosław Piliszek <email address hidden>
Date: Tue Sep 21 07:40:58 2021 +0000

    [debian] Use upstream MariaDB

    This preserves the Debian-provided minor version - 10.5 in case
    of Bullseye in Wallaby and Xena - and is backported as
    a fix to Victoria and Ussuri (where it follows Buster and its
    10.3).

    Closes-Bug: #1944410
    Change-Id: I9704e5371a410cb1f69ec0367174e2b368961dc1
    (cherry picked from commit b5422a6bc27797144da1ea973cec24a5cf46d2eb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/810744
Committed: https://opendev.org/openstack/kolla/commit/fe7b663235df3b924438cfe10c97d5f1f21e7226
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fe7b663235df3b924438cfe10c97d5f1f21e7226
Author: Radosław Piliszek <email address hidden>
Date: Tue Sep 21 07:40:58 2021 +0000

    [debian] Use upstream MariaDB

    This preserves the Debian-provided minor version - 10.5 in case
    of Bullseye in Wallaby and Xena - and will be backported as
    a fix to Victoria and Ussuri (where it will follow Buster and its
    10.3).

    Related-Bug: #1944410
    Change-Id: I9704e5371a410cb1f69ec0367174e2b368961dc1
    (cherry picked from commit b5422a6bc27797144da1ea973cec24a5cf46d2eb)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 10.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 11.2.0 release.