Unexpected API Error: unexpected keyword argument 'prompt'

[Jason Cornett]

When I try to launch a new instance I get "ERROR: Unable to create the server" in the GUI. I see the following in the nova API log:
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions [req-35c4e150-9b66-4fd3-96d0-8ff0f65d6174 00e8dfeb4681438d886a64071707a1c2 613f792787e1474f8982f2595a8a4766 - - -] Unexpected exception in API method
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 629, in create
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions **create_kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1562, in create
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1145, in _create_instance
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions reservation_id, max_count)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 872, in _validate_and_build_base_options
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions pci_request_info, requested_networks)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1064, in create_pci_requests_for_sriov_ports
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 139, in get_client
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 114, in _load_auth_plugin
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions auth_plugin = ks_loading.load_auth_from_conf_options(conf, NEUTRON_GROUP)
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/loading/conf.py", line 127, in load_from_conf_options
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions plugin_opts = plugin.get_options()
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneauth1/loading/_plugins/identity/v3.py", line 53, in get_options
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions help="User's password"),
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions 'plural': plural})
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions TypeError: __init__() got an unexpected keyword argument 'prompt'
2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
2016-08-05 15:05:54.110 8704 INFO nova.api.openstack.wsgi [req-35c4e150-9b66-4fd3-96d0-8ff0f65d6174 00e8dfeb4681438d886a64071707a1c2 613f792787e1474f8982f2595a8a4766 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2016-08-05 15:05:54.111 8704 INFO nova.osapi_compute.wsgi.server [req-35c4e150-9b66-4fd3-96d0-8ff0f65d6174 00e8dfeb4681438d886a64071707a1c2 613f792787e1474f8982f2595a8a4766 - - -] 10.7.41.159 "POST /v2/613f792787e1474f8982f2595a8a4766/servers HTTP/1.1" status: 500 len: 424 time: 0.7035410

[Augustina Ragwitz (auggy)]

Hi Jason, can you provide a little more information so I can attempt to reproduce this? What version/s are you running? Is this installed through Devstack or some other setup?

[Jason Cornett (jason-cornett1)]

Augustina,

I actually just wiped out the environment yesterday and went back to
Liberty due to multiple issues I was having. This was a fresh deployment of
Mitaka on CentOS 7. I deployed is using packstack as an all-in-one
controller. Let me know if you need anymore info.

Thanks,

Jason

On Tue, Aug 9, 2016 at 2:24 PM, Augustina Ragwitz <
<email address hidden>> wrote:

> Hi Jason, can you provide a little more information so I can attempt to
> reproduce this? What version/s are you running? Is this installed
> through Devstack or some other setup?
>
> ** Changed in: nova
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1610393
>
> Title:
> Unexpected API Error: unexpected keyword argument 'prompt'
>
> Status in OpenStack Compute (nova):
> Incomplete
>
> Bug description:
> When I try to launch a new instance I get "ERROR: Unable to create the
> server" in the GUI. I see the following in the nova API log:
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> [req-35c4e150-9b66-4fd3-96d0-8ff0f65d6174 00e8dfeb4681438d886a64071707a1c2
> 613f792787e1474f8982f2595a8a4766 - - -] Unexpected exception in API method
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> Traceback (most recent call last):
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line
> 478, in wrapped
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> return f(*args, **kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line
> 73, in wrapper
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> return func(*args, **kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line
> 73, in wrapper
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> return func(*args, **kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line
> 73, in wrapper
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> return func(*args, **kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py",
> line 629, in create
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> **create_kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions rv
> = f(*args, **kwargs)
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1562, in
> create
> 2016-08-05 15:05:54.107 8704 ERROR nova.api.openstack.extensions
> check_server_gro...

[Ken'ichi Ohmichi (oomichi)]

Humm, this problem seems strange for me.
The parameter "prompt" is added to both the caller side and the method side in a single patch: https://review.openstack.org/#/c/338614/

So based on this, "unexpected keyword argument" cannot happen.

[Shunde Zhang (shunde-zhang)]

Hi,
This happened to me too.

My observation is, it happens if I install keystone 9.1.0 and run refstack-client to test nova with 2016.01 list, but it works on the same environment if I run Rally certification tests. There isn't such problem if I install keystone 9.0.2. I tried using different versions of nova (13.0.0 and 13.1.1) but that didn't make any difference. So I suspect the cause is the new keystone.

I can try other combinations if required.

Hope this helps.

[Sean Dague (sdague)]

We need a pip freeze on the nova environment

[Pär Berge (paer-berge)]

Running RDO all-in-one on two CentOS 7 servers.

Having this issue on one of them maybe I can help debug this if needed?

At first glance I see this difference

# The working RDO
# rpm -q centos-release-openstack-mitaka
centos-release-openstack-mitaka-1-3.el7.noarch

# The one with this problem
rpm -q centos-release-openstack-mitaka
centos-release-openstack-mitaka-1-5.el7.noarch

[Pär Berge (paer-berge)]

Updated all packages and restarted, this solved the issue for me. I stored the output of rpm -qa before and after upgrade if that is of any value

[Rohanil Raje (rohanil)]

I am facing same issue with https://repos.fedorapeople.org/repos/openstack/openstack-mitaka/rdo-release-mitaka-5.noarch.rpm
How to fix it? If downgrading keystone is the fix, then how to do it? I am not able to do it using pip install keystone==9.0.2

Current versions are:
keystone==9.2.0
nova==13.1.2

[Artem Plakunov (artacc)]

I also face this problem but it happens very rarely with pretty much no pattern or ways to reproduce.

For me it happens when I launch instances through python api, it does not seem to matter how many at a time

It just happened again for 4 instances, but the next ones I launched a few minutes after were fine

environment: mirantis openstack 9.0 mitaka
keystone (9.0.2)
nova (13.0.0)
python-keystoneclient (2.3.1)
python-novaclient (3.3.1)

2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager [req-7d61c782-98cd-4585-ad55-bcade5c1d0f1 4862ee253d2f432890b4b2295dc29963 593dae96c84c405ca5eb91f4f1737a61 - - -] Instance failed network setup after 1 attempt(s)
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager Traceback (most recent call last):
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1570, in _allocate_network_async
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager bind_host_id=bind_host_id)
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 554, in allocate_for_instance
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager get_client(context, admin=True))
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 139, in get_client
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager _ADMIN_AUTH = _load_auth_plugin(CONF)
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 114, in _load_auth_plugin
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager auth_plugin = ks_loading.load_auth_from_conf_options(conf, NEUTRON_GROUP)
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/conf.py", line 127, in load_from_conf_options
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager plugin_opts = plugin.get_options()
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/keystoneauth1/loading/_plugins/identity/v3.py", line 53, in get_options
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager help="User's password"),
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager 'plural': plural})
2016-12-08 07:30:55.063 7553 ERROR nova.compute.manager TypeError: __init__() got an unexpected keyword argument 'prompt'
2016-12-08 07:30:55.267 7553 INFO nova.virt.libvirt.driver [req-7d61c782-98cd-4585-ad55-bcade5c1d0f1 4862ee253d2f432890b4b2295dc29963 593dae96c84c405ca5eb91f4f1737a61 - - -] [instance: 66a5e4ed-a444-429a-b4ff-db2470dc716e] Creating image
2016-12-08 07:30:55.726 7553 INFO nova.virt.libvirt.driver [req-7d61c782-98cd-4585-ad55-bcade5c1d0f1 4862ee253d2f432890b4b2295dc29963 593dae96c84c405ca5eb91f4f1737a61 - - -] [instance: 66a5e4ed-a444-429a-b4ff-db2470dc716e] Using config drive
2016-12-08 07:30:55.752 7553 ERROR nova.compute.manager [req...

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]