upgrade to Train might fail due to mariadb row format

[Radosław Piliszek]

In kolla-ansible CI we started getting on Ubuntu for Stein->Train:
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
for "ALTER TABLE instances ADD hidden BOOL" in nova upgrade

This is due to the following:

...
If the table is using either the REDUNDANT or the COMPACT row format, then one potential solution to this problem is to convert the table to use the DYNAMIC row format instead.

If your tables were originally created on an older version of MariaDB or MySQL, then your table may be using one of InnoDB's older row formats:

    In MariaDB 10.1 and before, and in MySQL 5.6 and before, the COMPACT row format was the default row format.
...
from https://mariadb.com/kb/en/library/troubleshooting-row-size-too-large-errors-with-innodb/#solving-the-problem

Indeed, the Stein images for Ubuntu include MariaDB 10.1 which we had to upgrade in Train due to other incompatibilities (with neutron).

It is also worth noting that old CentOS deployments may also have this limited row format.

[Radosław Piliszek (yoctozepto)]

Letting nova know too.

[Radosław Piliszek (yoctozepto)]

(touch screen + launchpad = random updates)

[Mark Goddard (mgoddard)]

FWIW, a solution proposed by the above mariadb page is:

ALTER TABLE tab ROW_FORMAT=DYNAMIC;

[Mark Goddard (mgoddard)]

A bit more context on the error:

http://paste.openstack.org/show/787539/

[Radosław Piliszek (yoctozepto)]

Yup, and we can find all with:

SELECT NAME, ROW_FORMAT
FROM information_schema.INNODB_SYS_TABLES
WHERE ROW_FORMAT IN('Redundant', 'Compact')
AND NAME NOT IN('SYS_DATAFILES', 'SYS_FOREIGN', 'SYS_FOREIGN_COLS', 'SYS_TABLESPACES', 'SYS_VIRTUAL', 'SYS_ZIP_DICT', 'SYS_ZIP_DICT_COLS');

[Matt Riedemann (mriedem)]

I don't think there is really anything for nova to do with this, it's dependent on how the database is configured correct?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/698957

[Radosław Piliszek (yoctozepto)]

@Matt, yes, though the old row format was the default. You can probably at most take this into account for docs (and when people come complaining outside of kolla).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698957
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4b21e938079589653437c6f857e4d400acd326e6
Submitter: Zuul
Branch: stable/train

commit 4b21e938079589653437c6f857e4d400acd326e6
Author: Radosław Piliszek <email address hidden>
Date: Fri Dec 13 17:38:03 2019 +0100

    Fix upgrades from older (<=10.1) MariaDB releases

    Due to old row format pre-upgrade some current migrations
    may fail to apply.
    In Train this affects only Ubuntu.
    This patch runs a script to change all row formats to DYNAMIC
    which is the default in recent MariaDB releases.

    This patch incorporates the following change to pass CI:

    CI: Set zuul's ansible_python_interpreter to python2

    Train, Stein & Rocky only

    Zuul seems to have started setting ansible_python_interpreter to python3
    on Ubuntu, which breaks our installation of kolla-ansible on Train and
    earlier stable branches.

    This change sets ansible_python_interpreter explicitly to python2. We
    use host-vars rather than vars because zuul sets a host-var for
    ansible_python_interpreter=auto, which overrides vars.

    I9ba0d16972b31a64617461461fafaf2b436e2bdf by Mark Goddard

    Change-Id: I3b39ec8d733b52a13cbc22b6f426b000584bab9d
    Closes-bug: #1856296

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc3 release candidate.

[alpha23 (alpha23)]

How long is the task, 'mariadb: Change row format to DYNAMIC for legacy tables' expected to take? On kolla-ansible 9.3.2, this process has already taken several hours and is still running.

[alpha23 (alpha23)]

It's actually been running for over 5 hours now. Horizon is inaccessible making the production Stein installation essentially unusable at this point, most likely because Horizon has not yet been upgraded to Train yet mariadb has been upgraded to train (per the docker tags).

As a note, prechecks passed without issue.

[alpha23 (alpha23)]

The result after ~6 hours was, " fatal: [<HOST>]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}"

UPDATE, the issue is that the query to alter the table does not handle tables which have dash. Result, e.g., is

"non-zero return code", "rc": 1, "start": "2022-01-15 18:27:40.980536", "stderr": "ERROR 1146 (42S02) at line 1: Table 'murano.environment@002dtemplate' doesn't exist", "stderr_lines": ["ERROR 1146 (42S02) at line 1: Table 'murano.environment@002dtemplate' doesn't exist"], "stdout": "", "stdout_lines": []}

I reported the same in a separate defect for train, https://bugs.launchpad.net/kolla-ansible/+bug/1958048

May affect other releases.