ValueError on creating new nova instance

[Tobias Urdin]

New bug introduced when upgrading nova-api on Ubuntu 14.04 for cloud archive liberty stable repo.
(deb http://ubuntu-cloud.archive.canonical.com/ubuntu trusty-updates/liberty main)

2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions [req-e3cb4d1c-a286-4450-aaa2-066072d1f306 3eb3f6111be04a8a957d8a3e8cb2dd86 6470df6b106e47508ebb00db08b557cf - - -] Unexpected exception in API method
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions **create_kwargs)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 149, in inner
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1587, in create
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1203, in _create_instance
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions block_device_mapping, legacy_bdm)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 862, in _check_and_transform_bdm
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions context, block_device_mapping)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/nova/objects/block_device.py", line 314, in block_device_make_list_from_dicts
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions for bdm in bdm_dicts_list]
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 294, in __init__
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions setattr(self, key, kwargs[key])
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 71, in setter
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions field_value = field.coerce(self, name, value)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 189, in coerce
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions return self._type.coerce(obj, attr, value)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/fields.py", line 304, in coerce
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions return int(value)
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions ValueError: invalid literal for int() with base 10: ''
2016-11-23 11:31:15.341 1600 ERROR nova.api.openstack.extensions
2016-11-23 11:31:15.361 1600 INFO nova.api.openstack.wsgi [req-e3cb4d1c-a286-4450-aaa2-066072d1f306 3eb3f6111be04a8a957d8a3e8cb2dd86 6470df6b106e47508ebb00db08b557cf - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.ValueError'>

[Tobias Urdin (tobias-urdin)]

Booting a nova instance with a block device as root volume.

[Tobias Urdin (tobias-urdin)]

ii nova-api 2:12.0.5-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-common 2:12.0.5-0ubuntu1~cloud0 all OpenStack Compute - common files
ii python-nova 2:12.0.5-0ubuntu1~cloud0 all OpenStack Compute Python libraries
ii python-novaclient 2:2.30.1-1~cloud0 all client library for OpenStack Compute API

[Tobias Urdin (tobias-urdin)]

No other changes than a package upgrade was made.

[Tobias Urdin (tobias-urdin)]

Only happens on the compute nodes running the previous version of nova, after upgrade instances could be spawned as usual.

As suggested by James on IRC:
11:53 < jamespage> tobasco, I suspect some sort of break on the RPC compat looking at the stacktrace but I might be wrong

Upgrade: nova-compute-libvirt:amd64 (12.0.4-0ubuntu1~cloud1, 12.0.5-0ubuntu1~cloud0)
Upgrade was done from 12.0.4 to 12.0.5

[Tobias Urdin (tobias-urdin)]

Not yet confirmed, checking with our product team if this could be our infrastructure sending a faulty API request. All our compute nodes is now updated thought so this issue might not be a bug.

Will update with more information once I have some...

[Tobias Urdin (tobias-urdin)]

Was caused by our internal infrastructure sending an invalid API request. I'm sorry for the hassle, marking as invalid.