Cannot delete volume after Cinder is upgraded to Newton

[Nam]

Cannot delete volume after Cinder is upgraded from Mitaka to Newton

How to reproduce:
I had three controller to configure high availibility (active/passive) for Cinder.

Topology: http://paste.openstack.org/show/588227/

Step 1: When Cinder is Mitaka release, creating a volume:

$ cinder create 1 --name volume_mitaka

Step 2: Start to upgrade Cinder by using the rolling-upgrade feature

During upgrading Cinder. Delete the volume_mitaka volume:
$ cinder delete volume_mitaka
Our expectation: Cinder can delete the volume but Cinder cannot delete it.

Step 3: After all component in Cinder were upgraded to Newton, continue to delete the volume:

$ cinder delete volume_mitaka

Our expectation: Cinder can delete the volume but Cinder cannot delete it.

Here is log in Cinder-api: http://paste.openstack.org/show/588239/
The result of the "cinder list" command: http://paste.openstack.org/show/588240/
On horizon: http://prnt.sc/d45r0q

[Michal Dulko (michal-dulko-f)]

Is the request dropped on the cinder-api level? Are there nothing in cinder-volume logs?

[Sean McGinnis (sean-mcginnis)]

Need the c-vol and c-sched logs to see what's going on.

And please clarify what you mean by "during upgrading Cinder".

[Nam (namnh)]

> And please clarify what you mean by "during upgrading Cinder".

As you know, currently Cinder supported the "rolling-upgrade" feature. "during upgrading Cinder", I mean that at that time, I finished upgrading cinder-volume to Newton release, cinder-api and cinder-scheduler are still Mitaka release. I cannot delete the volumes which I created before upgrading

One more thing, I am trying to get log from cinder volume as your request. I found a message in cinder-volume:

2016-11-09 13:41:29.618 ERROR oslo_messaging.rpc.server [-] Exception during message handling
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 157, in dispatch
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server raise UnsupportedVersion(version, method=method)
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server UnsupportedVersion: Endpoint does not support RPC version 1.40. Attempted method: delete_volume
2016-11-09 13:41:29.618 TRACE oslo_messaging.rpc.server

Maybe, I missed to configure in cinder.conf when starting cinder-volume. By the way, do you know the option in the cinder.conf file to pin version RPC?

[Michal Dulko (michal-dulko-f)]

Okay, so I'm pretty sure this was caused by old, orphaned records in ``services`` table in the DB. I think Nam was able to resolve the problem after some discussion on IRC [1].

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2016-11-08.log.html#t2016-11-08T08:42:50

[Nam (namnh)]

Thanks for your dicussion. I have just verified this case again. we can delete the volumes as normally during upgrading.