ncc upgrade to N or O release fail with db migration error
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Nova Cloud Controller Charm |
Fix Released
|
Critical
|
Felipe Reyes |
Bug Description
While testing openstack upgrade from M to N and O release, the ncc after the upgrade gets stuck at "db migration" and eventually (it takes about 20mins) errors to:
nova-cloud-
Also noticed that the none leader ncc fails to start haproxy and memcached, which blocks the service:
nova-cloud-
I have hit this issue on every try. All charms were upgraded to 17.02 release prior to openstack upgrade.
The juju-unit logs throws the following traceback:
2017-08-11 00:25:41 INFO juju-log shared-db:14: Migrating the nova database.
2017-08-11 00:25:45 DEBUG shared-
2017-08-11 00:25:45 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
2017-08-11 00:25:46 DEBUG shared-
Charm release version:
Stdout: |
commit-sha-1: 8c0168c9c37b97f
commit-short: 8c0168c
branch: HEAD
remote: https:/
info-generated: Thu Jul 6 19:28:20 UTC 2017
note: This file should exist only in a built or released charm artifact (not in the charm source code tree).
UnitId: nova-cloud-
sosreport from the affected unit is attached
description: | updated |
Changed in charm-nova-cloud-controller: | |
assignee: | nobody → Felipe Reyes (freyes) |
Changed in charm-nova-cloud-controller: | |
importance: | Undecided → High |
status: | New → Triaged |
milestone: | none → 17.08 |
Changed in charm-nova-cloud-controller: | |
importance: | High → Critical |
Changed in charm-nova-cloud-controller: | |
status: | Fix Committed → Fix Released |
On a recent test, the nova-schedule.log had the following error:
2017-08-23 17:07:22.616 1151495 ERROR nova guru_meditation _report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports. default_ filters" from group "DEFAULT" is deprecated. Use option "enabled_filters" from group "filter_scheduler". f4d2-49b5- 9b77-2fbd62ff7a 45 - - - - -] ProgrammingError: (pymysql. err.Programming Error) (1146, u"Table 'nova_api. aggregates' doesn't exist") [SQL: u'SELECT aggregates. created_ at AS aggregates_ created_ at, aggregates. updated_ at AS aggregates_ updated_ at, aggregates.id AS aggregates_id, aggregates.uuid AS aggregates_uuid, aggregates.name AS aggregates_name, aggregate_ hosts_1. created_ at AS aggregate_ hosts_1_ created_ at, aggregate_ hosts_1. updated_ at AS aggregate_ hosts_1_ updated_ at, aggregate_ hosts_1. id AS aggregate_ hosts_1_ id, aggregate_ hosts_1. host AS aggregate_ hosts_1_ host, aggregate_ hosts_1. aggregate_ id AS aggregate_ hosts_1_ aggregate_ id, aggregate_ metadata_ 1.created_ at AS aggregate_ metadata_ 1_created_ at, aggregate_ metadata_ 1.updated_ at AS aggregate_ metadata_ 1_updated_ at, aggregate_ metadata_ 1.id AS aggregate_ metadata_ 1_id, aggregate_ metadata_ 1.`key` AS aggregate_ metadata_ 1_key, aggregate_ metadata_ 1.value AS aggregate_ metadata_ 1_value, aggregate_ metadata_ 1.aggregate_ id AS aggregate_ metadata_ 1_aggregate_ id \nFROM aggregates LEFT OUTER JOIN aggregate_hosts AS aggregate_hosts_1 ON aggregates.id = aggregate_ hosts_1. aggregate_ id LEFT OUTER JOIN aggregate_metadata AS aggregate_ metadata_ 1 ON aggregates.id = aggregate_ metadata_ 1.aggregate_ id'] nova-scheduler" , line 10, in <module> python2. 7/dist- packages/ nova/cmd/ scheduler. py", line 44, in main scheduler_ topic) python2. 7/dist- packages/ nova/service. py", line 241, in create interval_ max=periodic_ interval_ max) python2. 7/dist- packages/ nova/service. py", line 117, in __init__ class(host= self.host, *args, **kwargs) python2. 7/dist- packages/ nova/scheduler/ manager. py", line 57, in __init__ on_load= True).driver python2. 7/dist- packages/ stevedore/ driver. py", line 61, in __init__ missing_ entrypoint= warn_on_ missing_ entrypoi. ..
2017-08-23 17:07:27.726 1151526 WARNING oslo_reports.
2017-08-23 17:07:27.746 1151526 WARNING oslo_config.cfg [-] Option "scheduler_
2017-08-23 17:07:28.070 1151526 CRITICAL nova [req-cf2e56c6-
2017-08-23 17:07:28.070 1151526 ERROR nova Traceback (most recent call last):
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/bin/
2017-08-23 17:07:28.070 1151526 ERROR nova sys.exit(main())
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/lib/
2017-08-23 17:07:28.070 1151526 ERROR nova topic=CONF.
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/lib/
2017-08-23 17:07:28.070 1151526 ERROR nova periodic_
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/lib/
2017-08-23 17:07:28.070 1151526 ERROR nova self.manager = manager_
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/lib/
2017-08-23 17:07:28.070 1151526 ERROR nova invoke_
2017-08-23 17:07:28.070 1151526 ERROR nova File "/usr/lib/
2017-08-23 17:07:28.070 1151526 ERROR nova warn_on_