Unable to launch an instance Bug 1219890

[mike_84OpenS]

I am unable to launch an instance.

[root@controller keystone]# nova boot --flavor m1.tiny --image cirros --nic net-id=544bcd85-b051-4e42-a462-d7dab712de5a --security-group default --key-name mykey public-instance

# ERRORs I am seeing:
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.BadRequest'> (HTTP 500) (Request-ID: req-5830ff32-d3e1-43a1-a3f9-7c6f96fb360b)

==> keystone/keystone.log <==
2016-07-29 22:03:07.865 2175 INFO keystone.common.wsgi [req-7e612649-5070-49f7-87da-71afe5f1c51c - - - - -] GET http://controller:5000/v3/
2016-07-29 22:03:07.980 2177 INFO keystone.common.wsgi [req-a6eda653-c25a-4453-b2e5-9736450f7568 - - - - -] POST http://controller:5000/v3/auth/tokens
2016-07-29 22:03:08.034 2198 INFO keystone.common.wsgi [req-4f9430cd-7579-489e-9bc7-c5cb3f477d5a - - - - -] GET http://controller:35357/v3/auth/tokens
2016-07-29 22:03:37.773 2186 INFO keystone.common.wsgi [req-8964cb8d-ea9c-4e7e-b153-4146c4e5b031 - - - - -] GET http://controller:5000/v3/
2016-07-29 22:03:37.778 2176 INFO keystone.common.wsgi [req-31e52a1e-ac13-42d9-92c5-9deea759e9e9 - - - - -] POST http://controller:5000/v3/auth/tokens
2016-07-29 22:03:37.828 2194 INFO keystone.common.wsgi [req-336a2901-5827-4508-829f-9de92ebd28da - - - - -] GET http://controller:35357/v3/auth/tokens
2016-07-29 22:03:37.961 2203 INFO keystone.common.wsgi [req-eb3d31c9-0aa3-4572-b78a-e087ae509de9 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-07-29 22:03:37.985 2200 INFO keystone.common.wsgi [req-ef7ab927-e5b3-45c0-89f9-d7191dc4cea4 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-07-29 22:03:38.120 2197 INFO keystone.common.wsgi [req-1ceee11a-cc13-490a-96ba-47081c1fe154 - - - - -] GET http://controller:35357/v3/auth/tokens
2016-07-29 22:03:38.174 2178 INFO keystone.common.wsgi [req-ebe3d994-18c2-4304-91a8-04143ba6a627 - - - - -] POST http://localhost:5000/v2.0/tokens
2016-07-29 22:03:38.174 2178 WARNING keystone.common.wsgi [req-ebe3d994-18c2-4304-91a8-04143ba6a627 - - - - -] Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

==> nova/nova-scheduler.log <==
[root@controller nova]# tail -1 nova-scheduler.log
2016-07-29 22:06:01.868 1409 INFO nova.scheduler.host_manager [req-1360cc28-fd91-42cb-918b-37db9432cc77 - - - - -] Successfully synced instances from host 'compute.example.com'.

==> nova/nova-api.log <==
2016-07-29 22:05:12.728 8504 INFO nova.osapi_compute.wsgi.server [req-54a6644f-96fe-4663-9825-ebc0e1ac920b 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/ HTTP/1.1" status: 200 len: 572 time: 0.0156450
2016-07-29 22:05:12.898 8504 INFO nova.osapi_compute.wsgi.server [req-d1ad4413-4c34-4a58-8b30-8dd258a8c9b1 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/7f659e1816d24cb2b2fdacfef970cfc6/images HTTP/1.1" status: 200 len: 692 time: 0.0624750
2016-07-29 22:05:12.964 8504 INFO nova.osapi_compute.wsgi.server [req-d62a91fe-c059-46e9-9084-e6294c714d5c 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/7f659e1816d24cb2b2fdacfef970cfc6/images/ffc86274-9a08-4b13-a23d-37c49bedb818 HTTP/1.1" status: 200 len: 873 time: 0.0643079
2016-07-29 22:05:12.979 8504 INFO nova.api.openstack.wsgi [req-e6f20077-6da7-4423-bb04-791b8eb1db2c 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] HTTP exception thrown: Flavor m1.tiny could not be found.
2016-07-29 22:05:12.980 8504 INFO nova.osapi_compute.wsgi.server [req-e6f20077-6da7-4423-bb04-791b8eb1db2c 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/7f659e1816d24cb2b2fdacfef970cfc6/flavors/m1.tiny HTTP/1.1" status: 404 len: 298 time: 0.0130160
2016-07-29 22:05:12.997 8504 INFO nova.osapi_compute.wsgi.server [req-c7db45cc-4338-4903-b97b-4672b96ac337 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/7f659e1816d24cb2b2fdacfef970cfc6/flavors?is_public=None HTTP/1.1" status: 200 len: 1407 time: 0.0142021
2016-07-29 22:05:13.012 8504 INFO nova.osapi_compute.wsgi.server [req-d82760c1-a126-4bbc-b599-5fa2a7d06bea 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "GET /v2/7f659e1816d24cb2b2fdacfef970cfc6/flavors/1 HTTP/1.1" status: 200 len: 615 time: 0.0121601
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions [req-5830ff32-d3e1-43a1-a3f9-7c6f96fb360b 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] Unexpected exception in API method
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions **create_kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1587, in create
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1187, in _create_instance
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions auto_disk_config, reservation_id, max_count)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 961, in _validate_and_build_base_options
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions pci_request_info, requested_networks)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1092, in create_pci_requests_for_sriov_ports
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions neutron = get_client(context, admin=True)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return self.get_access(session).auth_token
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions self.auth_ref = self.get_auth_ref(session)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions authenticated=False, log=False)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return self.request(url, 'POST', **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions raise exceptions.from_response(resp, method, url)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-1891a2c5-7edb-4a0f-a154-094803910893)
2016-07-29 22:05:13.123 8504 ERROR nova.api.openstack.extensions
2016-07-29 22:05:13.123 8504 INFO nova.api.openstack.wsgi [req-5830ff32-d3e1-43a1-a3f9-7c6f96fb360b 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
2016-07-29 22:05:13.124 8504 INFO nova.osapi_compute.wsgi.server [req-5830ff32-d3e1-43a1-a3f9-7c6f96fb360b 5d7190fb0b224c3484be7854426c88b1 7f659e1816d24cb2b2fdacfef970cfc6 - - -] 10.0.0.11 "POST /v2/7f659e1816d24cb2b2fdacfef970cfc6/servers HTTP/1.1" status: 500 len: 441 time: 0.1101460

OS Version - Liberty, Centos7
[root@controller nova]# rpm -qa | grep nova
openstack-nova-console-12.0.4-1.el7.noarch
openstack-nova-scheduler-12.0.4-1.el7.noarch
python-nova-12.0.4-1.el7.noarch
openstack-nova-novncproxy-12.0.4-1.el7.noarch
openstack-nova-conductor-12.0.4-1.el7.noarch
openstack-nova-common-12.0.4-1.el7.noarch
openstack-nova-cert-12.0.4-1.el7.noarch
openstack-nova-api-12.0.4-1.el7.noarch
python-novaclient-2.30.1-1.el7.noarch

Hypervisor: KVM.

[eil397 (anton-haldin)]

Hi mike_84OpenS
thank you for information.

Your case looks very close to https://bugs.launchpad.net/nova/+bug/1535079 for me.
Looks like documentation http://docs.openstack.org/liberty/install-guide-rdo/nova-controller-install.html was updated. Have you configured section [keystone_authtoken] ?

And you have mentioned https://bugzilla.redhat.com/show_bug.cgi?id=1219890. This bug was created for kilo.

Can you please provide configs for neutron and nova services ?

[mike_84OpenS (mike84open)]

Hi,

I have checked given bug and still not works. Here is my config:
<controller>
%%nova%%
[root@controller nova]# grep -v -e ^# -e ^$ /etc/nova/nova.conf
[DEFAULT]
url = http://controller:9696
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
username = neutron
password = password
service_metadata_proxy = True
metadata_proxy_shared_secret = password
enabled_apis=osapi_compute,metadata
network_api_class = nova.network.neutronv2.api.API
security_group_api = neutron
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
my_ip=10.0.0.11
auth_strategy=keystone
rpc_backend=rabbit
[api_database]
[barbican]
[cells]
[cinder]
[conductor]
[cors]
[cors.subdomain]
[database]
connection = mysql://nova:password@controller/nova
[ephemeral_storage_encryption]
[glance]
host=controller
[guestfs]
[hyperv]
[image_file_url]
[ironic]
[keymgr]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = password
[libvirt]
[matchmaker_redis]
[matchmaker_ring]
[metrics]
[neutron]
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host=controller
rabbit_userid=openstack
rabbit_password=password
[oslo_middleware]
[rdp]
[serial_console]
[spice]
[ssl]
[trusted_computing]
[upgrade_levels]
[vmware]
[vnc]
vncserver_listen = $my_ip
vncserver_proxyclient_address = $my_ip
[workarounds]
[xenserver]
[zookeeper]

%%neutron%%
[root@controller nova]# grep -v -e ^# -e ^$ /etc/neutron/neutron.conf
[DEFAULT]
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
core_plugin = ml2
service_plugins = router
allow_overlapping_ips = True
rpc_backend = rabbit
auth_strategy = keystone
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = password
[database]
connection = mysql://neutron:password@controller/neutron
[nova]
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
username = nova
password = password
[oslo_concurrency]
lock_path = /var/lib/neutron/tmp
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = password
[qos]

####################################################################################
<compute>
%%nova%%
[root@controller nova]# ssh compute 'grep -v -e ^# -e ^$ /etc/nova/nova.conf'
[DEFAULT]
network_api_class = nova.network.neutronv2.api.API
security_group_api = neutron
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
firewall_drive...

[mike_84OpenS (mike84open)]

Any suggestions, how to fix this ?

[Markus Zoeller (markus_z) (mzoeller)]

That's a configuration issue and not a bug in the nova code base. You can give this a try: https://bugs.launchpad.net/nova/+bug/1534273/comments/8

[vipin kumar (vipinwipl)]

90% liberty users have same issue, it is not getting away after applying change in nova.conf and other file as suggestion.