kilo default cinder behavior changed

[David Medberry]

Prior to Kilo, not having a:
[cinder]
os_region_name=SET

set meant that nova would look in the region the nova call went to. As of Kilo, it appears to do an endpoint list and pull the first cinder endpoint it finds (even if it is in a different region.)

I've not yet reproduced this in devstack but a multi-region devstack should see the right behavior prior to kilo and the wrong behavior in kilo (and presumably in Liberty.)

This has a VERY STRONG OPERATIONS IMPACT due to the difficulty in debugging a cross region error like this (it's extraordinarily opaque).

The simplest reproducer is to have a multi region openstack (for both nova and cinder) and create instances and volumes in both regions. In each region, try and attach a volume to a running instance. One side (whichever is uuid-numerically-lower) will likely succeed and the one that is numerically higher will fail (as it will try and find the volume in the wrong region.

The workaround is straight-forward--set
[cinder]
os_region_name=REGION
in the nova.conf file but as this is a behavior change, it should have been called out in the release notes.

Running:
openstack kilo 2015.1.0 (nova and cinder) in a multi (well two) region env.

distro info:

[STAGING] medberry@chrcnc02-control-003:~$ dpkg -l \*nova\* \*cinder\* \*keystone\* \*neutron\* |cat |grep ii
ii cinder-api 1:2015.1.0-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-backup 1:2015.1.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii cinder-common 1:2015.1.0-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 1:2015.1.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 1:2015.1.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
ii neutron-common 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-plugin-openvswitch-agent 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii neutron-server 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - server
ii nova-api 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - API frontend
ii nova-cert 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - certificate management
ii nova-common 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - common files
ii nova-conductor 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-objectstore 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - object store
ii nova-scheduler 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-cinder 1:2015.1.0-0ubuntu1~cloud0 all Cinder Python libraries
ii python-cinderclient 1:1.1.1-0ubuntu1~cloud0 all python bindings to the OpenStack Volume API
ii python-keystone 1:2015.1.0-0ubuntu1~cloud0 all OpenStack identity service - Python library
ii python-keystoneclient 1:1.2.0-0ubuntu1~cloud0 all Client library for OpenStack Identity API
ii python-keystonemiddleware 1.5.0-0ubuntu1~cloud0 all Middleware for OpenStack Identity (Keystone) - Python 2.x
ii python-neutron 1:2015.1.0-0ubuntu1~cloud0 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 2015.1.0-0ubuntu1~cloud0 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lbaas 1:2015.1.0-0ubuntu1~cloud0 all Loadbalancer-as-a-Service driver for OpenStack Neutron
ii python-neutronclient 1:2.3.11-0ubuntu1~cloud0 all client - Neutron is a virtual network service for Openstack
ii python-nova 1:2015.1.0-0ubuntu1.1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0 all client library for OpenStack Compute API
[STAGING] medberry@chrcnc02-control-003:~$

[Davanum Srinivas (DIMS) (dims-v)]

Can you please share the service catalog fields for both cinder and both nova entries?

[David Medberry (med)]

DIMS

dnvrco01-control-001.tgz is "NCW"

and

chrcnc01-control-001.tgz is region "NCE"

cinder.conf and nova.conf for each (with slight cleanup)

[David Medberry (med)]

DIMS

dnvrco01-control-001.tgz is "NCW"

and

chrcnc01-control-001.tgz is region "NCE"

cinder.conf and nova.conf for each (with slight cleanup)

[David Medberry (med)]

DIMS,

This is our current Juno deployment. Using this for Kilo means that volumes can't be attached (as it seems to use a non-region specific keystone endpoint list to determine where the volume comes from.) In our case (2 regions), one will succeed in regression tests, and one will fail. The one that succeeds is random (based on endpoint id as near as I can tell). The one that fails has a higher endpoint id for cinder and fails as it selects the lower.

This works fine in Juno but fails in Kilo.

Poke "med_" in #openstack-nova IRC if you need more info (or update the bug).

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]