Liberty -> Mitaka Upgrade: Error: Unable to create the server. Unexpected API Error.

[Tytus Kurek]

Description
===========
After upgrading fresh OpenStack Liberty installation to Mitaka on Trusty with Juju, I am no longer able to create Nova instances.

Steps to reproduce
==================
* Install OpenStack Liberty with Juju from the attached "bundle.yaml" file.
* Upgrade OpenStack Liberty to Mitaka by executing the following commands:

juju upgrade-charm keystone
juju upgrade-charm ceilometer
juju upgrade-charm ceilometer-agent
juju upgrade-charm ceph
juju upgrade-charm ceph-osd
juju upgrade-charm cinder
juju upgrade-charm glance
juju upgrade-charm nova-cloud-controller
juju upgrade-charm nova-compute
juju upgrade-charm neutron-api
juju upgrade-charm neutron-gateway
juju upgrade-charm openstack-dashboard
juju set-config ceph source="cloud:trusty-mitaka"
juju set-config ceph-osd source="cloud:trusty-mitaka"
juju set-config keystone openstack-origin="cloud:trusty-mitaka"
juju ssh keystone/0 sudo keystone-manage db_sync
juju set-config ceilometer openstack-origin="cloud:trusty-mitaka"
juju set-config ceilometer-agent openstack-origin="cloud:trusty-mitaka"
juju set-config cinder openstack-origin="cloud:trusty-mitaka"
juju ssh cinder/0 sudo cinder-manage db sync
juju set-config glance openstack-origin="cloud:trusty-mitaka"
juju ssh glance/0 sudo glance-manage db_sync
juju set-config nova-cloud-controller openstack-origin="cloud:trusty-mitaka"
juju ssh nova-cloud-controller/0 sudo nova-manage db sync
juju set-config nova-compute openstack-origin="cloud:trusty-mitaka"
juju set-config neutron-api openstack-origin="cloud:trusty-mitaka"
juju ssh neutron-api/0 sudo neutron-db-manage upgrade heads
juju set-config neutron-gateway openstack-origin="cloud:trusty-mitaka"
juju set-config openstack-dashboard openstack-origin="cloud:trusty-mitaka"
juju ssh ceph/0 sudo reboot
juju ssh ceph/1 sudo reboot
juju ssh ceph/2 sudo reboot
juju ssh ceph-osd/0 sudo service ceph restart
juju ssh ceph-osd/1 sudo service ceph restart
juju ssh ceph-osd/2 sudo service ceph restart
juju ssh keystone/0 sudo reboot
juju ssh keystone/1 sudo reboot
juju ssh keystone/2 sudo reboot
juju ssh ceilometer/0 sudo reboot
juju ssh ceilometer/1 sudo reboot
juju ssh ceilometer/2 sudo reboot
juju ssh ceilometer-agent/0 sudo service ceilometer-agent-compute restart
juju ssh ceilometer-agent/1 sudo service ceilometer-agent-compute restart
juju ssh ceilometer-agent/2 sudo service ceilometer-agent-compute restart
juju ssh cinder/0 sudo reboot
juju ssh cinder/1 sudo reboot
juju ssh cinder/2 sudo reboot
juju ssh glance/0 sudo reboot
juju ssh glance/1 sudo reboot
juju ssh glance/2 sudo reboot
juju ssh nova-cloud-controller/0 sudo reboot
juju ssh nova-cloud-controller/1 sudo reboot
juju ssh nova-cloud-controller/2 sudo reboot
juju ssh nova-compute/0 sudo service nova-compute restart
juju ssh nova-compute/1 sudo service nova-compute restart
juju ssh nova-compute/2 sudo service nova-compute restart
juju ssh neutron-api/0 sudo reboot
juju ssh neutron-api/1 sudo reboot
juju ssh neutron-api/2 sudo reboot
juju ssh neutron-gateway/0 sudo service neutron-dhcp-agent restart
juju ssh neutron-gateway/0 sudo service neutron-lbaas-agent restart
juju ssh neutron-gateway/0 sudo service neutron-metadata-agent restart
juju ssh neutron-gateway/0 sudo service neutron-metering-agent restart
juju ssh neutron-gateway/0 sudo service neutron-openvswitch-agent restart
juju ssh neutron-gateway/0 sudo service neutron-vpn-agent restart
juju ssh nova-compute/0 sudo service neutron-openvswitch-agent restart
juju ssh nova-compute/1 sudo service neutron-openvswitch-agent restart
juju ssh nova-compute/2 sudo service neutron-openvswitch-agent restart
juju ssh openstack-dashboard/0 sudo reboot
juju ssh openstack-dashboard/1 sudo reboot
juju ssh openstack-dashboard/2 sudo reboot

* Attempt to create Nova instance.

Expected result
===============
Nova instance being created.

Actual result
=============
* Nova instance not being created.
* The following error messages being displayed:
** from GUI:

   Error: Unable to create the server.

** from CLI:

   Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
   <type 'exceptions.AttributeError'> (HTTP 500) (Request-ID: req-c035d518-50c0-4cab-913e-9a5263392a2a)

Environment
===========
1. Exact version of OpenStack you are running.

ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l | grep nova
ii nova-api-os-compute 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - OpenStack Compute API frontend
ii nova-cert 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - certificate management
ii nova-common 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-conductor 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-consoleauth 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - Console Authenticator
ii nova-novncproxy 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python-nova 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Connection to 10.230.126.81 closed.
ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep nova
ii nova-common 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2~cloud0 all client library for OpenStack Compute API - Python 2.7
Connection to 10.230.126.54 closed.

2. Which hypervisor did you use?

KVM

   What's the version of that?

ubuntu@tkurek-maas:~$ juju ssh nova-compute/0 sudo dpkg-query -l | grep kvm
ii nova-compute-kvm 2:13.1.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
Connection to 10.230.126.54 closed.

3. Which storage type did you use?

Ceph

   What's the version of that?

ubuntu@tkurek-maas:~$ juju ssh ceph/0 sudo dpkg-query -l | grep ceph
ii ceph 10.2.2-0ubuntu0.16.04.2~clo amd64 distributed storage and file system
ii ceph-common 10.2.2-0ubuntu0.16.04.2~clo amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-0ubuntu0.16.04.2~clo amd64 Ceph distributed file system client library
ii python-ceph 10.2.2-0ubuntu0.16.04.2~clo all Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.2-0ubuntu0.16.04.2~clo amd64 Python libraries for the Ceph libcephfs library
Connection to 10.230.126.65 closed.
ubuntu@tkurek-maas:~$ juju ssh ceph-osd/0 sudo dpkg-query -l | grep ceph
ii ceph 10.2.2-0ubuntu0.16.04.2~clo amd64 distributed storage and file system
ii ceph-common 10.2.2-0ubuntu0.16.04.2~clo amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-0ubuntu0.16.04.2~clo amd64 Ceph distributed file system client library
ii python-ceph 10.2.2-0ubuntu0.16.04.2~clo all Meta-package for python libraries for the Ceph libraries
ii python-cephfs 10.2.2-0ubuntu0.16.04.2~clo amd64 Python libraries for the Ceph libcephfs library
Connection to 10.230.126.54 closed.

4. Which networking type did you use?

Neutron with Openvswitch

5. Additional information:

ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg -S /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
python-dogpile.cache: /usr/lib/python2.7/dist-packages/dogpile/cache/region.py
Connection to 10.230.126.81 closed.
ubuntu@tkurek-maas:~$ juju ssh nova-cloud-controller/0 sudo dpkg-query -l python-dogpile.cache
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=============================================-===========================-===========================-===============================================================================================
ii python-dogpile.cache 0.5.7-1ubuntu1~cloud0 all caching front-end based on the Dogpile lock - Python 2.x
Connection to 10.230.126.81 closed.

Logs & Configs
==============
Attached "/var/log/nova/nova-api-os-compute.log" file errors:

2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions [req-b2f38657-7eae-4714-a1e1-0217950e7e29 d12595ae6b2d418eaf9ffd209fa25479 64c87895e12b4f4fb0bfd979b1ac70a4 - - -] Unexpected exception in API method
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions **create_kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1562, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1175, in _create_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota, filter_properties)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1033, in _provision_instances
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions quotas.rollback()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self.force_reraise()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 988, in _provision_instances
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions num_instances, i, shutdown_terminate)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1445, in create_db_entry_for_new_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance.create()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 471, in create
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self._load_ec2_ids()
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 818, in _load_ec2_ids
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions self.ec2_ids = objects.EC2Ids.get_by_instance(self._context, self)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 181, in wrapper
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 218, in get_by_instance
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions ec2_ids = cls._get_ec2_ids(context, instance)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/ec2.py", line 204, in _get_ec2_ids
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions instance.image_ref)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 128, in glance_id_to_ec2_id
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions image_id = glance_id_to_id(context, glance_id)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/ec2/ec2utils.py", line 48, in memoizer
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions value = _CACHE.get(key)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/cache_utils.py", line 151, in get
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions value = self.region.get(key)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 456, in get
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions expiration_time, ignore_expiration)(value)
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/dogpile/cache/region.py", line 475, in value_fn
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions current_time - value.metadata["ct"] > expiration_time:
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions AttributeError: 'int' object has no attribute 'metadata'
2016-08-20 11:03:48.780 558458 ERROR nova.api.openstack.extensions

[Sean Dague (sdague)]

You are going to need to unwind the juju bundle into something the dev team understands, or poke the juju folks.

[Tytus Kurek (tkurek)]

The cloud has already gone, but the issue is reproducible. How can I poke juju folks?

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Sam Morrison (sorrison)]

Reopening this bug, going through an upgrade from liberty -> mitaka and getting this bug

[Sam Morrison (sorrison)]

Happy to provide more details, we are using stable/mitaka version of conductor with the rest still at liberty

dogpile.cache==0.6.2

[Saverio Proto (zioproto)]

Hello,
I am hitting this bug as well.

We dont use Juju, but we also run the same ubuntu packages for trusty

[Saverio Proto (zioproto)]

Hello Sam,

I managed to fix the bug.

It happens when you have liberty stuff in nova.conf that you dont need anymore.

First test I did was to copy the nova.conf.dpkg-dist to nova.conf and then run puppet. This fixed the problem immediately.

Then I tracked down what was the line giving the problem.

Make sure you remove from nova.conf the memcached_servers options form the [DEFAULT] section, having it only in the [keystone_authtoken] fixed this problem for me.

[Sean Dague (sdague)]

Apparently this is a misconfiguration issue with memcached_servers. Can people post their nova.conf to understand what went wrong here? If we need to add a warning on that option it would be good to know.

[Sean Dague (sdague)]

Automatically discovered version mitaka in description. If this is incorrect, please update the description to include 'nova version: ...'

[Sean Dague (sdague)]

Did this give a deprecation warning on Liberty? Or did we just miss that one entirely?

[Sean Dague (sdague)]

I'm marking this incomplete, it would be good to get the answer, however these branches are so outside support there really isn't an upstream fix.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]