upgrade to Train might fail due to mariadb row format

Bug #1856296 reported by Radosław Piliszek
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Undecided
Unassigned
kolla-ansible
Invalid
Undecided
Unassigned
Train
Fix Released
Critical
Radosław Piliszek
Ussuri
Invalid
Undecided
Unassigned

Bug Description

In kolla-ansible CI we started getting on Ubuntu for Stein->Train:
Row size too large. The maximum row size for the used table type, not counting BLOBs, is 8126. This includes storage overhead, check the manual. You have to change some columns to TEXT or BLOBs
for "ALTER TABLE instances ADD hidden BOOL" in nova upgrade

This is due to the following:

...
If the table is using either the REDUNDANT or the COMPACT row format, then one potential solution to this problem is to convert the table to use the DYNAMIC row format instead.

If your tables were originally created on an older version of MariaDB or MySQL, then your table may be using one of InnoDB's older row formats:

    In MariaDB 10.1 and before, and in MySQL 5.6 and before, the COMPACT row format was the default row format.
...
from https://mariadb.com/kb/en/library/troubleshooting-row-size-too-large-errors-with-innodb/#solving-the-problem

Indeed, the Stein images for Ubuntu include MariaDB 10.1 which we had to upgrade in Train due to other incompatibilities (with neutron).

It is also worth noting that old CentOS deployments may also have this limited row format.

Tags: mariadb nova
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Letting nova know too.

description: updated
Changed in kolla-ansible:
status: Triaged → Won't Fix
status: Won't Fix → Triaged
Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

(touch screen + launchpad = random updates)

Revision history for this message
Mark Goddard (mgoddard) wrote :

FWIW, a solution proposed by the above mariadb page is:

ALTER TABLE tab ROW_FORMAT=DYNAMIC;

Revision history for this message
Mark Goddard (mgoddard) wrote :

A bit more context on the error:

http://paste.openstack.org/show/787539/

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

Yup, and we can find all with:

SELECT NAME, ROW_FORMAT
FROM information_schema.INNODB_SYS_TABLES
WHERE ROW_FORMAT IN('Redundant', 'Compact')
AND NAME NOT IN('SYS_DATAFILES', 'SYS_FOREIGN', 'SYS_FOREIGN_COLS', 'SYS_TABLESPACES', 'SYS_VIRTUAL', 'SYS_ZIP_DICT', 'SYS_ZIP_DICT_COLS');

Revision history for this message
Matt Riedemann (mriedem) wrote :

I don't think there is really anything for nova to do with this, it's dependent on how the database is configured correct?

Changed in nova:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to kolla-ansible (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/698957

Revision history for this message
Radosław Piliszek (yoctozepto) wrote :

@Matt, yes, though the old row format was the default. You can probably at most take this into account for docs (and when people come complaining outside of kolla).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla-ansible (stable/train)

Reviewed: https://review.opendev.org/698957
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=4b21e938079589653437c6f857e4d400acd326e6
Submitter: Zuul
Branch: stable/train

commit 4b21e938079589653437c6f857e4d400acd326e6
Author: Radosław Piliszek <email address hidden>
Date: Fri Dec 13 17:38:03 2019 +0100

    Fix upgrades from older (<=10.1) MariaDB releases

    Due to old row format pre-upgrade some current migrations
    may fail to apply.
    In Train this affects only Ubuntu.
    This patch runs a script to change all row formats to DYNAMIC
    which is the default in recent MariaDB releases.

    This patch incorporates the following change to pass CI:

    CI: Set zuul's ansible_python_interpreter to python2

    Train, Stein & Rocky only

    Zuul seems to have started setting ansible_python_interpreter to python3
    on Ubuntu, which breaks our installation of kolla-ansible on Train and
    earlier stable branches.

    This change sets ansible_python_interpreter explicitly to python2. We
    use host-vars rather than vars because zuul sets a host-var for
    ansible_python_interpreter=auto, which overrides vars.

    I9ba0d16972b31a64617461461fafaf2b436e2bdf by Mark Goddard

    Change-Id: I3b39ec8d733b52a13cbc22b6f426b000584bab9d
    Closes-bug: #1856296

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla-ansible 9.0.0.0rc3

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc3 release candidate.

Revision history for this message
alpha23 (alpha23) wrote :

How long is the task, 'mariadb: Change row format to DYNAMIC for legacy tables' expected to take? On kolla-ansible 9.3.2, this process has already taken several hours and is still running.

Revision history for this message
alpha23 (alpha23) wrote :

It's actually been running for over 5 hours now. Horizon is inaccessible making the production Stein installation essentially unusable at this point, most likely because Horizon has not yet been upgraded to Train yet mariadb has been upgraded to train (per the docker tags).

As a note, prechecks passed without issue.

Revision history for this message
alpha23 (alpha23) wrote (last edit ):

The result after ~6 hours was, " fatal: [<HOST>]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true}"

UPDATE, the issue is that the query to alter the table does not handle tables which have dash. Result, e.g., is

"non-zero return code", "rc": 1, "start": "2022-01-15 18:27:40.980536", "stderr": "ERROR 1146 (42S02) at line 1: Table 'murano.environment@002dtemplate' doesn't exist", "stderr_lines": ["ERROR 1146 (42S02) at line 1: Table 'murano.environment@002dtemplate' doesn't exist"], "stdout": "", "stdout_lines": []}

I reported the same in a separate defect for train, https://bugs.launchpad.net/kolla-ansible/+bug/1958048

May affect other releases.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers