Upgrade to Train fails while running Cinder online schema migration

[Pierre Riteau]

During an upgrade from Stein to Train, Kolla Ansible fails while running TASK [cinder : Running Cinder online schema migration]

This is because the `--max_count 10` option is used, which returns 1 while migrations are processed. According to the upgrade documentation, the command should be rerun while the exit status is 1: https://docs.openstack.org/cinder/train/upgrade.html

  msg: Container exited with non-zero return code 1
  rc: 1
  stderr: |-
    + sudo -E kolla_set_configs
    INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
    INFO:__main__:Validating config file
    INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
    INFO:__main__:Copying service configuration files
    INFO:__main__:Deleting /etc/cinder/cinder.conf
    INFO:__main__:Copying /var/lib/kolla/config_files/cinder.conf to /etc/cinder/cinder.conf
    INFO:__main__:Setting permission for /etc/cinder/cinder.conf
    INFO:__main__:Copying /var/lib/kolla/config_files/cinder-wsgi.conf to /etc/httpd/conf.d/cinder-wsgi.conf
    INFO:__main__:Setting permission for /etc/httpd/conf.d/cinder-wsgi.conf
    INFO:__main__:Writing out command to execute
    INFO:__main__:Setting permission for /var/lib/cinder
    INFO:__main__:Setting permission for /var/lib/cinder/tmp
    INFO:__main__:Setting permission for /var/log/kolla/cinder
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.2.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.5.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-volume.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.2.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-manage.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.6.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.3.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.1
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.1
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.6.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.3.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-wsgi.log.4.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-scheduler.log
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.4.gz
    INFO:__main__:Setting permission for /var/log/kolla/cinder/cinder-api-access.log.5.gz
    ++ cat /run_command
    + CMD='httpd -DFOREGROUND'
    + ARGS=
    + sudo kolla_copy_cacerts
    + [[ ! -n '' ]]
    + . kolla_extend_start
    ++ [[ ! -d /var/log/kolla/cinder ]]
    +++ stat -c %a /var/log/kolla/cinder
    ++ [[ 755 != \7\5\5 ]]
    ++ . /usr/local/bin/kolla_cinder_extend_start
    +++ set -o errexit
    +++ [[ -n '' ]]
    +++ [[ -n 0 ]]
    +++ [[ -n 0 ]]
    +++ cinder-manage db online_data_migrations --max_count 10
    Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
  stderr_lines: <omitted>
  stdout: |-
    10 rows matched query untyped_volumes_online_data_migration, 10 migrated
    +-----------------------------------------+--------------+-----------+
    | Migration | Total Needed | Completed |
    +-----------------------------------------+--------------+-----------+
    | untyped_volumes_online_data_migration | 10 | 10 |
    +-----------------------------------------+--------------+-----------+
  stdout_lines: <omitted>

**Environment**:

* CentOS 7.8
* Kolla-Ansible version: stable/train
* Docker image Install type (source/binary): binary
* Docker image distribution: centos
* Official images from Docker Hub

[Pierre Riteau (priteau)]

Workaround:

- set cinder_max_number_osm to a high number
- run upgrade: it will fail again, but will have processed many more migrations
- run upgrade again, which should work since cinder-manage will return 0

[Mark Goddard (mgoddard)]

I suspect this was inadvertently caused by https://review.opendev.org/#/c/712055/. While the logic was incorrect previously, it caused us to ignore the max count. Probably the right solution is to stop passing a max count through from kolla-ansible. Alternatively we could add a loop in kolla, but what would we do between iterations? sleep?

[Pierre Riteau (priteau)]

It's not clear in what circumstances it is important to use a custom max_count. Note that if max_count is not passed to cinder-manage, it will run batches of 50 migrations until complete (i.e. only one invocation should be needed).

If someone wants to implement the loop logic, I would suggest defaulting to no sleep but allowing to customize it via a variable. But if no one really needs to use that feature, let's stop passing max_count.

[Radosław Piliszek (yoctozepto)]

I opt for not using this variable there.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/735617

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/735617
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca
Submitter: Zuul
Branch: master

commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/736316

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/736317

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/736318

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736316
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=851180d26661e2cc2a5dc5ec06f22bb1f11efb87
Submitter: Zuul
Branch: stable/ussuri

commit 851180d26661e2cc2a5dc5ec06f22bb1f11efb87
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736317
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=6f227c2cdf4590790616210fa94340594822c4d4
Submitter: Zuul
Branch: stable/train

commit 6f227c2cdf4590790616210fa94340594822c4d4
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/736318
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=041943fa1c4b25ace589728abb909fe6624acd8e
Submitter: Zuul
Branch: stable/stein

commit 041943fa1c4b25ace589728abb909fe6624acd8e
Author: Mark Goddard <email address hidden>
Date: Mon Jun 15 16:19:49 2020 +0100

    Remove max count from Cinder online schema migration

    During an upgrade from Stein to Train, Kolla Ansible fails while running
    TASK [cinder : Running Cinder online schema migration]

    This is because the `--max_count 10` option is used, which returns 1
    while migrations are processed. According to the upgrade documentation,
    the command should be rerun while the exit status is 1:
    https://docs.openstack.org/cinder/train/upgrade.html

    This issue was introduced by a change to the image [1] which fixed a bug
    in the way that the max count was interpreted, but exposed an issue in
    using the max count.

    This change fixes the issue by ceasing to pass MAX_NUMBER, which will
    cause all migrations to occur in a single pass.

    [1] https://review.opendev.org/#/c/712055

    Change-Id: Ia786d037f5484f18294188639c956d4ed5ffbc2a
    Closes-Bug: #1880753
    (cherry picked from commit 55c0787d00d678b2fbfc9e99a6e4b4ce5c9a82ca)