oslo_messaging.rpc.client.RemoteError upon server add volume

[flashydave]

Description
===========
from openstack cli:
 server add volume <server> <volume>
fails to add volume but returns with:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'oslo_messaging.rpc.client.RemoteError'> (HTTP 500) (Request-ID: req-bfe801f2-c3b3-4f17-9b21-42e2e397ff14)

Steps to reproduce
==================
1/ Create server from image
2/ create bare volume
3/ server add volume <server> <volume>

Expected result
===============
Volume attached

Actual result
=============
Exception as above

Environment
===========
openstack pike stable

pip list | grep kolla
kolla 6.0.0.0rc1
kolla-ansible 6.0.0

ceph being used used for cinder storage.

[jichenjc (jichenjc)]

are you sure the kolla setup is correct?
checked the log of api seems

RemoteError: Remote error: RemoteError Remote error: IncompatibleObjectVersion Version 2.4 of Instance is not supported

looks like v2.4 was added after Pike release, so I assume you might using master branch compute node of kolla ??

commit bc9449573a3503a177b8dacc6b15f5108df00309
Merge: 54dc704 589eb88
Author: Zuul <email address hidden>
Date: Tue Apr 3 15:54:50 2018 +0000

    Merge "Add trusted_certs object"

commit 589eb8872631ff4f0a4b34003152a644a35cddb7
Author: Jackie Truong <email address hidden>
Date: Fri Dec 15 19:01:00 2017 -0500

    Add trusted_certs object

1337890a (Dan Smith 2015-09-04 15:20:23 -0700 111) # Version 2.0: Initial version
b65a8794 (Tomi Juvonen 2016-01-15 09:20:36 +0200 112) # Version 2.1: Added services
e83842b8 (Dan Smith 2016-05-04 11:45:38 -0700 113) # Version 2.2: Added keypairs
1aa49410 (Vladik Romanovsky 2016-06-07 15:25:47 -0400 114) # Version 2.3: Added device_metadata
589eb887 (Jackie Truong 2017-12-15 19:01:00 -0500 115) # Version 2.4: Added trusted_certs
589eb887 (Jackie Truong 2017-12-15 19:01:00 -0500 116) VERSION = '2.4'

[Matt Riedemann (mriedem)]

Right there is no Instance 2.4 in stable/pike:

https://github.com/openstack/nova/blob/stable/pike/nova/objects/instance.py#L115

Did you backport some code to the Instance object in Pike?

What is this config option set to?

https://docs.openstack.org/nova/pike/configuration/config.html#upgrade_levels.compute

[upgrade_levels]
compute

[flashydave (dave-opensourcesolutions)]

Thanks for feedback. Maybe my understanding that pip installation of kolla and kolla-ansible at the versions stated constitute pike. Kolla's own docs seem to have changed since I originally created my system and in an attempt to fix a really problematic ceph issue (reinstallion of the rds following a major network failure) I may have upgraded using pip -U to something more recent. It's possible I have got burned somehow.

I havent installed from git even though I have the repos for reference whilst debugging.

I have upgrade_levels set to auto. Maybe I need to be more specific or better control which kolla versions I am running. Their most recent guide provide some warnings about PBR that I dont recall being there earlier.

[jichenjc (jichenjc)]

yes, you may consult with kolla folks, I mostly specified the openstack release version
in kolla globals.yml and no problem to me when I upgrade something..