os_nova: upgrade_levels/compute=auto failure on master

Bug #1753443 reported by git-harry
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
git-harry
Queens
In Progress
Medium
Matt Riedemann
OpenStack-Ansible
Fix Released
Undecided
Unassigned

Bug Description

It looks like a recent change [1] in nova, to remove RPC 4.x support, has exposed a bug when using upgrade_levels/compute=auto on a new deployment. This is blocking the openstack-ansible-os_nova master gate.

Tempest tests are failing, the following in nova-conductor.log shows the failure:
```
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server [req-9c3f2dd3-81dd-4275-9a61-a3a859dde29d 3639ea84ebcf4c858de98eeede6789a9 3b9624e03ed740f483c64301d0d11372 - default default] Exception during message handling: RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 4.11.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 1265, in schedule_and_build_instances
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server limits=host.limits, host_list=host_list)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 1030, in build_and_run_instance
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server cctxt.cast(ctxt, 'build_and_run_instance', **kwargs)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 149, in cast
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server self._check_version_cap(msg.get('version'))
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-testing/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 126, in _check_version_cap
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server version_cap=self.version_cap)
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 4.11.
2018-03-03 05:13:23.679 9771 ERROR oslo_messaging.rpc.server
```

When openstack-ansible-os_nova is used for a new deployment, the following appears in the logs:
```
2018-03-02 17:25:55.954 19495 DEBUG nova.compute.rpcapi [req-97c173ed-052e-4ce7-8314-d220dfdab8e7 - - - - -] Not caching compute RPC version_cap, because min service_version is 0. Please ensure a nova-compute service has been started. Defaulting to Mitaka RPC. _determine_version_cap /openstack/venvs/nova-master/lib/python2.7/site-packages/nova/compute/rpcapi.py:408
```

The reference to Mitaka is caused by [2], it looks to be intended to set the version cap to be as permissive as possible (N to N+1 upgrades) but it appears it hasn't been updated since it was first added for newton.

Restarting the services addresses the issue, observed by:
```
2018-03-04 21:42:14.367 21270 INFO nova.compute.rpcapi [req-95678c1e-8465-4059-8095-70479085b179 - - - - -] Automatically selected compute RPC version 5.0 from minimum service version 30
```

It seems like there may be two issues exposed here, one is the bug in nova setting the minimum version and the other is how OSA handles the deployment. With the default OSA deployment the minimum RPC version will change with a restart, it would seem that has the potential to cause failures if the order of the restarts is not controlled given those restarts are not triggered by the deployment process.

[1] https://github.com/openstack/nova/commit/a761e57368280b6d3e931831ecd393fd5787b3ef
[2] https://github.com/openstack/nova/blob/a761e57368280b6d3e931831ecd393fd5787b3ef/nova/compute/rpcapi.py#L384-L392

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/549737

Changed in nova:
assignee: nobody → git-harry (git-harry)
status: New → In Progress
new (cloudie)
Changed in nova:
assignee: git-harry (git-harry) → nobody
assignee: nobody → new (cloudie)
Changed in openstack-ansible:
status: New → In Progress
new (cloudie)
Changed in openstack-ansible:
status: In Progress → Fix Committed
status: Fix Committed → Fix Released
Changed in nova:
assignee: new (cloudie) → git-harry (git-harry)
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/549737
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b81a66b3b0b1ce9f56280dda40ac871132cb29f2
Submitter: Zuul
Branch: master

commit b81a66b3b0b1ce9f56280dda40ac871132cb29f2
Author: git-harry <email address hidden>
Date: Mon Mar 5 10:48:28 2018 +0000

    Fix version cap when no nova-compute started

    When a zero service version is returned, it means that we have no
    services running for the requested binary. In that case, we should
    assume the latest version available until told otherwise. This usually
    happens in first-start cases, where everything is likely to be up to
    date anyway.

    This change addresses an issue where the version returned had been
    hard-coded to 4.11 (mitaka).

    Change-Id: I696a8ea8adbe9481e11407ecafd5e47b2bd29804
    Closes-bug: 1753443

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/550557

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/queens)

Change abandoned by Matt Riedemann (<email address hidden>) on branch: stable/queens
Review: https://review.openstack.org/550557
Reason: not needed for queens since that still supports 4.x

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.0.0b1

This issue was fixed in the openstack/nova 18.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.