Upgrade from bionic-rocky to bionic-stein failed migrations.

[Angel Vargas]

We were trying to upgrade from rocky to stein using the charm procedure described here:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-upgrade-openstack.html

and we got into this problem,

2019-07-02 09:56:44 ERROR juju-log online_data_migrations failed
b'Running batches of 50 until complete\nError attempting to run <function fill_virtual_interface_list at 0x7fee41f77048>\n9 rows matched query populate_user_id, 0 migrated\n+---------------------------------------------+--------------+-----------+\n| Migration | Total Needed | Completed |\n+---------------------------------------------+--------------+-----------+\n| create_incomplete_consumers | 0 | 0 |\n| delete_build_requests_with_no_instance_uuid | 0 | 0 |\n| fill_virtual_interface_list | 0 | 0 |\n| migrate_empty_ratio | 0 | 0 |\n| migrate_keypairs_to_api_db | 0 | 0 |\n| migrate_quota_classes_to_api_db | 0 | 0 |\n| migrate_quota_limits_to_api_db | 0 | 0 |\n| migration_migrate_to_uuid | 0 | 0 |\n| populate_missing_availability_zones | 0 | 0 |\n| populate_queued_for_delete | 0 | 0 |\n| populate_user_id | 9 | 0 |\n| populate_uuids | 0 | 0 |\n| service_uuids_online_data_migration | 0 | 0 |\n+---------------------------------------------+--------------+-----------+\nSome migrations failed unexpectedly. Check log for details.\n'

What should we do to get this fixed?

Regards,

[Angel Vargas (angelvargas)]

After about 15 minutes the process said has ended, but the outcome is mess, as we lost all the resources control, as many resources are missing and I can not access the hypervisors

:~/master-node/openstack/juju-units-conf$ openstack hypervisor list
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'ModuleNotFoundError'> (HTTP 500) (Request-ID: req-e2ec7d6c-b6c2-4d6c-a52e-fae80fbea324)

[Sahid Orentino (sahid-ferdjaoui)]

Looks like an issue when executing 'nova-manager db online_data_migrations'.

Can you share package version currently installed:
$ dpkg -s nova-common

Also sharing logs of nova-manage located under /var/log/nova may help.

If you have not configured your env in debug, please consider using:
$ juju config nova-cloud-controller debug=true

Then you could try to ru-run locally:
$ nova-manager db online_data_migrations

Thanks,
s.

[Angel Vargas (angelvargas)]

Hi,

this is how I have solved the problem, seems after the upgrade most of the packages got upgraded to python3 but there was missing a package for mysql

Logs showed:

/home/ubuntu# cat /var/log/nova/nova-manage.log

", line 534, in _setup_for_connection
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage sql_connection=sql_connection, **engine_kwargs)
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage return wrapped(*args, **kwargs)
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 177, in create_engine
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage engine = sqlalchemy.create_engine(url, **engine_args)
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage File "/usr/lib/python3/dist-packages/sqlalchemy/engine/__init__.py", line 425, in create_engine
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage return strategy.create(*args, **kwargs)
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 81, in create
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage dbapi = dialect_cls.dbapi(**dbapi_args)
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage File "/usr/lib/python3/dist-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 102, in dbapi
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage return __import__('MySQLdb')
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage ModuleNotFoundError: No module named 'MySQLdb'
2019-07-02 10:01:14.453 729831 ERROR nova.cmd.manage
2019-07-02 10:01:14.463 729831 WARNING oslo_db.sqlalchemy.engines [req-42da38ea-f420-4e4c-ab58-e19d68804d10 - - - - -] URL mysql://nova:***@10.10.0.202/nova does not contain a '+drivername' portion, and will make use of a default driver. A full dbname+drivername:// protocol is recommended. For MySQL, it is strongly recommended that mysql+pymysql:// be specified for maximum service compatibility
2019-07-02 10:01:14.464 729831 WARNING nova.objects.instance_mapping [req-42da38ea-f420-4e4c-ab58-e19d68804d10 - - - - -] Encountered exception: "No module named 'MySQLdb'" while querying instances from cell: e5f8f404-9361-40ad-a67a-a0746d4868b0(cell1). Continuing to the next cell.: ModuleNotFoundError: No module named 'MySQLdb'

After see that in the logs, I checked and effectively there wasn't a lib for python - mysql

I did:

sudo apt-get install -y python3-mysqldb

and then all worked fine.

Is this a isolated case or is actually a bug? Why should be this package missing after the upgrade?

[Sahid Orentino (sahid-ferdjaoui)]

Thanks for your investigation and sharing resolution. I don't think it's isolated case I saw several issues on other charms around that.

[Angel Vargas (angelvargas)]

Sorry @sahid-ferdjaoui I missed your questions:

Re: Package version where we hitted the bug:

ubuntu@juju-45ee93-2-lxd-2:~$ dpkg -s nova-common
Package: nova-common
Status: install ok installed
Priority: extra
Section: net
Installed-Size: 505
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Source: nova
Version: 2:19.0.0-0ubuntu2~cloud0
Replaces: nova-ajax-console-proxy (<< 2:18.0.0~b2-0ubuntu3~), nova-api (<< 2:18.0.0~b2-0ubuntu3~), nova-api-metadata (<< 2:18.0.0~b2-0ubuntu3~), nova-api-os-compute (<< 2:18.0.0~b2-0ubuntu3~), nova-cells (<< 2:18.0.0~b2-0ubuntu3~), nova-compute (<< 2:18.0.0~b2-0ubuntu3~), nova-compute-libvirt (<< 2:18.0.0~b2-0ubuntu3~), nova-compute-vmware (<< 2:18.0.0~b2-0ubuntu3~), nova-conductor (<< 2:18.0.0~b2-0ubuntu3~), nova-console (<< 2:18.0.0~b2-0ubuntu3~), nova-consoleauth (<< 2:18.0.0~b2-0ubuntu3~), nova-network (<< 2:18.0.0~b2-0ubuntu3~), nova-novncproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-placement-api (<< 2:18.0.0~b2-0ubuntu3~), nova-scheduler (<< 2:18.0.0~b2-0ubuntu3~), nova-serialproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-spiceproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-xvpvncproxy (<< 2:18.0.0~b2-0ubuntu3~), python-nova (<< 2:18.0.0-0ubuntu2~)
Depends: adduser
Breaks: nova-ajax-console-proxy (<< 2:18.0.0~b2-0ubuntu3~), nova-api (<< 2:18.0.0~b2-0ubuntu3~), nova-api-metadata (<< 2:18.0.0~b2-0ubuntu3~), nova-api-os-compute (<< 2:18.0.0~b2-0ubuntu3~), nova-cells (<< 2:18.0.0~b2-0ubuntu3~), nova-compute (<< 2:18.0.0~b2-0ubuntu3~), nova-compute-libvirt (<< 2:18.0.0~b2-0ubuntu3~), nova-compute-vmware (<< 2:18.0.0~b2-0ubuntu3~), nova-conductor (<< 2:18.0.0~b2-0ubuntu3~), nova-console (<< 2:18.0.0~b2-0ubuntu3~), nova-consoleauth (<< 2:18.0.0~b2-0ubuntu3~), nova-network (<< 2:18.0.0~b2-0ubuntu3~), nova-novncproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-placement-api (<< 2:18.0.0~b2-0ubuntu3~), nova-scheduler (<< 2:18.0.0~b2-0ubuntu3~), nova-serialproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-spiceproxy (<< 2:18.0.0~b2-0ubuntu3~), nova-xvpvncproxy (<< 2:18.0.0~b2-0ubuntu3~), python-nova (<< 2:18.0.0-0ubuntu2~)
Conffiles:
 /etc/logrotate.d/nova-common c8b4e3ac8388c4395b7fb8af33068753
 /etc/nova/api-paste.ini adfc3bc985a189b27e45b7be66d1a296
 /etc/nova/logging.conf dd4cdd10a6db5e711c519067603dcec2
 /etc/nova/nova.conf 0473e0723f12e4b51c44fb3209ca0a35
 /etc/nova/rootwrap.conf 90396580aced374b1ac7e25b49a68bb1
 /etc/nova/rootwrap.d/api-metadata.filters 973935718801ea5e303bf167809b66f2
 /etc/nova/rootwrap.d/compute.filters 6b66ceadead04371d5437e3e11407851
 /etc/nova/rootwrap.d/network.filters 58e5367d0ed714b5780be81f7a004bb5
 /etc/sudoers.d/nova_sudoers e06b72d1c3653b1eca9747f8423a1d16
Description: OpenStack Compute - common files
 OpenStack is a reliable cloud infrastructure. Its mission is to produce
 the ubiquitous cloud computing platform that will meet the needs of public
 and private cloud providers regardless of size, by being simple to implement
 and massively scalable.
 .
 OpenStack Compute, codenamed Nova, is a cloud computing fabric controller. In
 addition to its "native" API (the OpenStack API), it also supports the Amazon
 EC2 API.
 .
 Nova is intended to be modular and easy to extend and adapt. I...

[James Page (james-page)]

At Stein, the default sqlalchemy mysql driver was switched back to mysqldb; prior to stein we just used the default which was mysqldb in older releases, and pymysql from at least queens/bionic->cosmic/rocky.

The pymysql package is installed, its just that the connection URL in the nova configuration file is not using the correct pymysql+ prefix to switch to the pymysql driver.

[James Page (james-page)]

Sahid and I chatted about this last night; the issue is that the DB urls written into the Nova DB for each cell only have the mysql:// prefix; with the switch of the default dialect to MySQLDB at stein, these need to be updated to be mysql+pymysql://.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/671252

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/671253

[Sahid Orentino (sahid-ferdjaoui)]

I also proposed a fix for nova since 'nova-manage cellv2 update_cell' is bugged for cell0.

  https://review.opendev.org/#/c/672045/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672321

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672321
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=3c806a23f6dada5b1bc4417099d328f97e5c7e2a
Submitter: Zuul
Branch: master

commit 3c806a23f6dada5b1bc4417099d328f97e5c7e2a
Author: James Page <email address hidden>
Date: Tue Jul 23 16:47:06 2019 +0100

    stein: install python3-mysqldb for sqlalchemy dialect change

    At Stein, the default mysql dialect in SQLAlchemy switched back
    to MySQLDB in Ubuntu; as a result the db connection URL's stored
    in the nova_api database for Nova cells are stale and need to
    prefixed mysql+pymysql:// to work with the pymysql dialect.

    However at this point in time there is no way to update the DB
    URL for Nova Cell 0.

    This fix works around this problem by installing python3-mysqldb
    which provides the default dialect needed when using mysql://
    prefixed DB connection URI's.

    Change-Id: I3f74f18f649786fefa113fc3d8fa7c24010a185d
    Related-Bug: 1835037

[Alvaro Uria (aluria)]

I've hit this same issue. See https://pastebin.ubuntu.com/p/hnwFBvhSnY/

[Ryan Beisner (1chb1n)]

The fix merged in master and is in the current stable charms as of 19.10.