nova-manage db online_data_migrations can fail with UnicodeEncodeError if there are duplicate aggregates or flavors

[Matt Riedemann]

This was found purely through code inspection, but the online data migration to move aggregates to the nova API database has a problem here:

https://github.com/openstack/nova/blob/a74d3ae4e815e3727961ef67bd801dada0267a0b/nova/objects/aggregate.py#L596

The problem is the AggregateNameExists exception message is translatable and could be in unicode characters, and when casting that to str() in python 2.7 it results in a UnicodeEncodeError:

Python 2.7.6 (default, Jun 22 2015, 18:00:18)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> msg = u'\xF0\x9F\x92\xA9'
>>> msg
u'\xf0\x9f\x92\xa9'
>>> str(msg)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)

[Matt Riedemann (mriedem)]

I've marked this as High severity because anyone that hits this is going to be blocked on running their online data migrations.

[Matt Riedemann (mriedem)]

Looks like this was copied from the online flavor migration code here:

https://github.com/openstack/nova/blob/a74d3ae4e815e3727961ef67bd801dada0267a0b/nova/objects/flavor.py#L726

[Matt Riedemann (mriedem)]

It also looks like hacking check N325 was meant for catching stuff like this but it doesn't handle the LOG.error(str(e)) format of this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415899

[Matt Riedemann (mriedem)]

Dropping the severity back to medium because I think it'd be pretty rare to hit this case as the aggregate and flavors object code checks to see if the main database is empty (migrated) before creating a new resource in the API DB so the only way I can think of that this duplicate exception would happen is if someone pre-populated the API database rather than run the online migrations, or if there was some sort of weird race failure during the migration. Plus you could workaround it by deleting the duplicate resource from the main DB.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415896
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=641f5f4529509debef6e8e58f01a2ea20109c0be
Submitter: Jenkins
Branch: master

commit 641f5f4529509debef6e8e58f01a2ea20109c0be
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 30 11:53:49 2016 -0500

    Actually test online flavor migrations

    The way the test was written for online flavor migrations
    we didn't have any flavors in the main database, so nothing
    was migrated and nothing was asserting that we had migrated
    anything. This is more or less due to using USES_DB_SELF
    and then not populating the main database with any flavors.

    This change creates a single flavor in the main database
    prior to the online data migration and asserts we actually
    started with one and it was migrated.

    Change-Id: Ia18f8abdf10785dc396f5c07e239326059528daf
    Partial-Bug: #1653261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415898
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eda54b1ed73b0c102f4352e1522e34908dc7c17f
Submitter: Jenkins
Branch: master

commit eda54b1ed73b0c102f4352e1522e34908dc7c17f
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 30 12:27:48 2016 -0500

    Handle unicode when dealing with duplicate flavors during online migrations

    In python 2.7 you can't cast unicode to str and you'll get a
    UnicodeEncodeError if you try. Since our exception messages are
    translated, they can contain unicode and if we hit a duplicate flavor
    exception while performing online flavor migrations it will break the
    migration and block any further migrations until resolved - which would
    require manual intervention.

    This patch fixes the bug so that we use six.text_type instead of str
    for logging the duplicate exception and adds a test to exhibit the bug
    and prove it's fixed.

    On a side note, it's curious that we don't delete the duplicate flavor
    from the main database during the online data migration, but it's also
    strange that you'd have duplicates in the API database because the
    Flavor.create() checks to see if all flavors have been migrated from
    the main DB to the API DB and if not you can't create new flavors in
    the API DB - so the only way to get the duplicates is either by inserting
    them into the API DB manually or perhaps some other kind of race issue.
    Anyway, that's not dealt with here.

    Change-Id: I3bdb1a8ca72c3e72ddc3bc5102cff8df18148617
    Partial-Bug: #1653261

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415899
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=539d3512d647ffe562091b7facbc287e75077d92
Submitter: Jenkins
Branch: master

commit 539d3512d647ffe562091b7facbc287e75077d92
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 30 12:56:20 2016 -0500

    Handle unicode when dealing with duplicate aggregate errors during migration

    The AggregateNameExists message can be translated and have unicode in it
    so we can't cast it to a str in python 2.7 else we'll get a
    UnicodeEncodeError. That would break the online data migration for
    aggregates along with any other online data migrations that run after
    that one which impacts upgrades.

    This uses six.text_type rather than str for logging the error and adds
    a test to recreate the bug and show that it's fixed.

    Change-Id: I040db22ecbb9fabe5bbda8a3d9600cc9f76cb170
    Closes-Bug: #1653261

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.0.0.0b3 development milestone.