Failed `nova-manage db sync` returns exitcode of 0

Bug #1538227 reported by Florian Ermisch on 2016-01-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Stephen Finucane

Bug Description

We're trying to upgrade to liberty using SaltStack
(just for context, same issue running the command
from a shell).

At one point `nova-manage db sync` is executed.
Because of #1511466 we get a *critical* error but
the command still returns an exitcode of 0.
In a somewhat POSIX environment this means
"everything is fine" so the deployment just
continues with an outdated database schema.

          ID: nova-manage db sync
    Function: cmd.run
        Name: nova-manage db sync; sleep 15
      Result: True
     Comment: Command "nova-manage db sync; sleep 15" run
     Started: 18:07:09.511897
    Duration: 17747.739 ms
     Changes:
              ----------
              pid:
                  18000
              retcode:
                  0
              stderr:
                  No handlers could be found for logger "oslo_config.cfg"
                  2016-01-26 18:07:12.092 18001 DEBUG migrate.versioning.repository [-] Loading repository /usr/lib/python2.7/dist-packages/nova/db/sql
alchemy/migrate_repo... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/repository.py:76
                  2016-01-26 18:07:12.093 18001 DEBUG migrate.versioning.script.base [-] Loading script /usr/lib/python2.7/dist-packages/nova/db/sqlalc
hemy/migrate_repo/versions/216_havana.py... __init__ /usr/lib/python2.7/dist-packages/migrate/versioning/script/base.py:27
[...]
[...]
[...]
                  2016-01-26 18:07:12.157 18001 INFO migrate.versioning.api [-] 290 -> 291...
                  2016-01-26 18:07:12.167 18001 CRITICAL nova [-] ValidationError: There are still 3 unmigrated flavor records. Migration cannot conti$
ue until all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                  2016-01-26 18:07:12.167 18001 ERROR nova Traceback (most recent call last):
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/bin/nova-manage", line 10, in <module>
                  2016-01-26 18:07:12.167 18001 ERROR nova sys.exit(main())
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1443, in main
                  2016-01-26 18:07:12.167 18001 ERROR nova ret = fn(*fn_args, **fn_kwargs)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 910, in sync
                  2016-01-26 18:07:12.167 18001 ERROR nova return migration.db_sync(version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                  2016-01-26 18:07:12.167 18001 ERROR nova return IMPL.db_sync(version=version, database=database)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_$
ync
                  2016-01-26 18:07:12.167 18001 ERROR nova version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "<string>", line 2, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in
with_engine
                  2016-01-26 18:07:12.167 18001 ERROR nova return f(*a, **kw)
                  2016-01-26 18:07:12.167 18001 ERROR nova return migration.db_sync(version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/migration.py", line 26, in db_sync
                  2016-01-26 18:07:12.167 18001 ERROR nova return IMPL.db_sync(version=version, database=database)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migration.py", line 106, in db_s
ync
                  2016-01-26 18:07:12.167 18001 ERROR nova version)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 186, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova return _migrate(url, repository, version, upgrade=True, err=err, **opts)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "<string>", line 2, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 160, in
with_engine
                  2016-01-26 18:07:12.167 18001 ERROR nova return f(*a, **kw)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 366, in _migrate
                  2016-01-26 18:07:12.167 18001 ERROR nova schema.runchange(ver, change, changeset.step)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 93, in runchan$
e
                  2016-01-26 18:07:12.167 18001 ERROR nova change.run(self.engine, step)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/migrate/versioning/script/py.py", line 148, in run
                  2016-01-26 18:07:12.167 18001 ERROR nova script_func(engine)
                  2016-01-26 18:07:12.167 18001 ERROR nova File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/migrate_repo/versions/291_enfor$
e_flavors_migrated.py", line 35, in upgrade
                  2016-01-26 18:07:12.167 18001 ERROR nova raise exception.ValidationError(detail=msg)
                  2016-01-26 18:07:12.167 18001 ERROR nova ValidationError: There are still 3 unmigrated flavor records. Migration cannot continue unt$
l all instance flavor records have been migrated to the new format. Please run `nova-manage db migrate_flavor_data' first.
                  2016-01-26 18:07:12.167 18001 ERROR nova
              stdout:
                  Command failed, please check log for more info
----------

root@controller:~# dpkg -l | grep nova
ii nova-api 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - API frontend
ii nova-cert 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - certificate management
ii nova-common 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - common files
ii nova-conductor 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - virtual machine scheduler
ii nova-spiceproxy 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute - spice html5 proxy
ii python-nova 2:12.0.0-0ubuntu2~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API
root@controller:~#

How to reproduce:
* setup openstack < liberty
* update pkgs to liberty
* run `nova-manage db sync && echo "you shouldn't see this because the prev cmd failed" `
(The error while running the command is topic of issue #1511466)

Expected result:
Failing `nova-manage db sync` returns a none-zero exitcode and thus signals a failure
as any utility on a unix-like system should.

Actual result:
Failing `nova-manage db sync` still returns an exitcode of 0 signaling successful
update of database schemas to the running shell/command/automation tool of choice.

Sean Dague (sdague) on 2016-02-17
Changed in nova:
status: New → Confirmed
importance: Undecided → High
tags: added: nova-manage
Changed in nova:
assignee: nobody → Stephen Finucane (sfinucan)

Fix proposed to branch: master
Review: https://review.openstack.org/289308

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/289308
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=173068d52449eb04e905b29da6e417ae69cd8055
Submitter: Jenkins
Branch: master

commit 173068d52449eb04e905b29da6e417ae69cd8055
Author: Stephen Finucane <email address hidden>
Date: Mon Mar 7 12:34:47 2016 +0000

    nova-manage: Print, not raise, exceptions

    Raising an exception means it is not possible to return a status code.
    Seeing as the 'nova-manage' application is an interactive one, it's
    viable to fix this by printing the exception rather than raising.

    Change-Id: Ifa3f80e4f7dccada439ffc9363e9f1504c8c2da1
    Closes-bug: #1538227

Changed in nova:
status: In Progress → Fix Released

Change abandoned by Stephen Finucane (<email address hidden>) on branch: master
Review: https://review.openstack.org/289309
Reason: There has to be a better way to do this

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers