OpenStack Nova Cloud Controller Charm

  1. Bug #1910276
  2. Activity log

Activity log for bug #1910276

Date Who What changed Old value New value Message
2021-01-05 18:54:07 Alex Kavanagh bug added bug
2021-01-05 18:54:17 Alex Kavanagh tags openstack-upgrade
2021-01-06 16:18:54 Alex Kavanagh charm-nova-compute: status New Invalid
2021-01-07 12:21:54 Alex Kavanagh charm-nova-compute: status Invalid New
2021-01-07 12:25:06 Alex Kavanagh bug task added charm-nova-cloud-controller
2021-01-07 12:25:12 Alex Kavanagh charm-nova-compute: status New Invalid
2021-01-07 15:16:14 Alex Kavanagh summary openstack-upgrade bionic:rocky -> stein -> train (starting clean) results in nova-compute not running openstack-upgrade bionic:stein -> train without placement and then adding placement results in a confusing state
2021-01-07 15:18:14 Alex Kavanagh description Juju 2.7.8-bionic-amd64 Test run from func-target: bionic-rocky and then manual upgrades to stein, then train (leaving rmq and percona-cluster on bionic-rocky. Result from /var/log/nova/nova-compute.log: 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service [req-26da2780-45c2-4a66-900d-4099d11c6d02 - - - - -] Error starting thread.: oslo_messaging.rpc.client.RemoteError: Remote error: IncompatibleObjectVersion Version 2.6 of InstanceList is not supported, supported version is 2.4 ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 141, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', 'oslo_versionedobjects.exception.IncompatibleObjectVersion: Version 2.6 of InstanceList is not supported, supported version is 2.4\n']. 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service Traceback (most recent call last): 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 810, in run_service 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service service.start() 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 174, in start 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service self.manager.init_host() 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1337, in init_host 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata', 'numa_topology']) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service args, kwargs) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service args=args, kwargs=kwargs) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 178, in call 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service retry=self.retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 127, in _send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service retry=retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service call_monitor_timeout, retry=retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 635, in _send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service raise result 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service oslo_messaging.rpc.client.RemoteError: Remote error: IncompatibleObjectVersion Version 2.6 of InstanceList is not supported, supported version is 2.4 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 141, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', 'oslo_versionedobjects.exception.IncompatibleObjectVersion: Version 2.6 of InstanceList is not supported, supported version is 2.4\n']. 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service See comment #4 below. Essentially, the charm can get to the state where the config option for openstack-origin doesn't match the installed payload version but due to placement being related to it, there is no user facing information that explains what the issue is. Makes you want a "juju status-report" that includes an analysis by the charm itself of issues on the charm. -- original below Juju 2.7.8-bionic-amd64 Test run from func-target: bionic-rocky and then manual upgrades to stein, then train (leaving rmq and percona-cluster on bionic-rocky. Result from /var/log/nova/nova-compute.log: 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service [req-26da2780-45c2-4a66-900d-4099d11c6d02 - - - - -] Error starting thread.: oslo_messaging.rpc.client.RemoteError: Remote error: IncompatibleObjectVersion Version 2.6 of InstanceList is not supported, supported version is 2.4 ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 141, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', 'oslo_versionedobjects.exception.IncompatibleObjectVersion: Version 2.6 of InstanceList is not supported, supported version is 2.4\n']. 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service Traceback (most recent call last): 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 810, in run_service 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service service.start() 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/service.py", line 174, in start 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service self.manager.init_host() 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 1337, in init_host 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service expected_attrs=['info_cache', 'metadata', 'numa_topology']) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service args, kwargs) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service args=args, kwargs=kwargs) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 178, in call 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service retry=self.retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 127, in _send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service retry=retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service call_monitor_timeout, retry=retry) 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 635, in _send 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service raise result 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service oslo_messaging.rpc.client.RemoteError: Remote error: IncompatibleObjectVersion Version 2.6 of InstanceList is not supported, supported version is 2.4 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service ['Traceback (most recent call last):\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 141, in object_class_action_versions\n objname, object_versions[objname])\n', ' File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 387, in obj_class_from_name\n supported=latest_ver)\n', 'oslo_versionedobjects.exception.IncompatibleObjectVersion: Version 2.6 of InstanceList is not supported, supported version is 2.4\n']. 2021-01-05 17:57:02.384 28521 ERROR oslo_service.service
2021-01-07 15:18:18 Alex Kavanagh charm-nova-cloud-controller: importance Undecided Medium
2021-01-07 20:15:56 Alex Kavanagh charm-nova-cloud-controller: assignee Alex Kavanagh (ajkavanagh)
2021-01-07 20:16:00 Alex Kavanagh charm-nova-cloud-controller: status New In Progress