undercloud install fails (nova-db-sync timeout) on VM on an SATA disk hypervisor

[Alex Schultz]

2017-02-01 15:24:49,084 INFO: Error: Command exceeded timeout
2017-02-01 15:24:49,084 INFO: Error: /Stage[main]/Nova::Db::Sync/Exec[nova-db-sync]/returns: change from notrun to 0 failed: Command exceeded timeout

The nova-db-sync command is exceeding 300 seconds when installing the undercloud on a VM that is using SATA based storage. This seems to be related to the switch to innodb_file_per_table to ON which has doubled the amount of time the db sync takes on this class of hardware. To unblock folks doing Ocata testing, we need to skip doing this in Ocata and will need to revisit enabling it in Pike.

See Bug 1660722 for details as to why we enabled this.

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/428435/

[Mike Bayer (zzzeek)]

why aren't we opening a bug in Nova that their migrations are overall exceedingly slow?

[Emilien Macchi (emilienm)]

Mike: done.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428435
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=3f7e74ab24bb43f9ad7e24e0efd4206ac6a3dd4e
Submitter: Jenkins
Branch: master

commit 3f7e74ab24bb43f9ad7e24e0efd4206ac6a3dd4e
Author: Alex Schultz <email address hidden>
Date: Thu Feb 2 21:29:32 2017 +0000

    Revert "set innodb_file_per_table to ON for MySQL / Galera"

    This reverts commit 621ea892a299d2029348db2b56fea1338bd41c48.

    We're getting performance problems on SATA disks.

    Change-Id: I30312fd5ca3405694d57e6a4ff98b490de388b92
    Closes-Bug: #1661396
    Related-Bug: #1660722

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/428843

[Matt Riedemann (mriedem)]

Do you have a log file or something that shows us which of the migrations are taking the longest so someone could dig into this?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428843
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=578599f2eed3611b66ed96031a8e6260716b4b79
Submitter: Jenkins
Branch: master

commit 578599f2eed3611b66ed96031a8e6260716b4b79
Author: Alex Schultz <email address hidden>
Date: Fri Feb 3 11:09:34 2017 -0700

    Increase sync timeout for nova db syncs

    We have seen on lower quality hardware that the nova db syncs can take
    an excessive amount of time. In order to still support deploying on this
    hardware, let's increase the timeout from the default 300 seconds to 900
    seconds to allow for this less performant gear. It is not recommended to
    increase this past 900 as if we start hitting this then we need to be
    understanding what is occuring in these db syncs. 300 seconds should be
    enough time to setup a database especially on install. But there are
    cases for upgrades or slower disks where it can exceed 300 seconds.

    Change-Id: I77507c638237072e38d9888aff3da884aeff0b59
    Related-Bug: #1660722
    Related-Bug: #1661396

[Matt Riedemann (mriedem)]

I wonder if https://review.openstack.org/#/c/430390/ would indirectly help here in case it makes the base (and biggest) DB schema migration in nova run faster.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 6.2.0 release.

[Sean Dague (sdague)]

Automatically discovered version ocata in description. If this is incorrect, please update the description to include 'nova version: ...'