Launch Instance got 'NeutronAdminCredentialConfigurationInvalid'

[Bino Oetomo]

I try to launch new instance (from horizon as 'admin').

From empty list of instance, click 'launch instance'.

Configuration windows poped,

After some configuration (selecting image etc etc) click the 'launch instance' (while watching /var/log/nova/nova-api.log) got this messages :

===============LAUNCH================

2019-02-08 15:56:16.138 19148 INFO nova.osapi_compute.wsgi.server [req-914666f9-5a2d-4323-9f92-515276f75565 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] 127.0.0.1 "GET /v2.1 HTTP/1.0" status: 302 len: 300 time: 3.1299040

2019-02-08 15:56:19.252 19148 INFO nova.osapi_compute.wsgi.server [req-a53933c1-f157-4c24-ab87-5f5b14839e84 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] 127.0.0.1 "GET /v2.1/ HTTP/1.0" status: 200 len: 771 time: 3.1098912

2019-02-08 15:56:39.069 19147 ERROR nova.network.neutronv2.api [req-bc73b885-bab4-4116-8497-96bbe782ea4d 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] Neutron client was not able to generate a valid admin token, please verify Neutron admin credential located in nova.conf: Unauthorized: 401-{u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}}
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi [req-bc73b885-bab4-4116-8497-96bbe782ea4d 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] Unexpected exception in API method: NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 110, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 576, in create
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi **create_kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 154, in inner
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi rv = f(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1714, in create
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi trusted_certs=trusted_certs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1161, in _create_instance
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi reservation_id, max_count)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 772, in _validate_and_build_base_options
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi self._check_requested_secgroups(context, security_groups))
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 382, in _check_requested_secgroups
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi secgroup_dict = self.security_group_api.get(context, secgroup)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/security_group/neutron_driver.py", line 143, in get
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi neutron, 'security_group', name, context.project_id)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 61, in find_resourceid_by_name_or_id
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi parent_id, fields='id')['id']
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/neutron/v2_0/__init__.py", line 53, in find_resource_by_name_or_id
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi cmd_resource, parent_id, fields)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 474, in find_resource
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi cmd_resource, parent_id, fields)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 430, in find_resource_by_id
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi data = obj_lister(**params)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 976, in list_security_groups
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi retrieve_all, **_params)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in list
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi for r in self._pagination(collection, path, **params):
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in _pagination
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi res = self.get(path, params=params)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 354, in get
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi headers=headers, params=params)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 331, in retry_request
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi headers=headers, params=params)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 126, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi ret = obj(*args, **kwargs)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi self._handle_fault_response(status_code, replybody, resp)
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 142, in wrapper
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi raise exception.NeutronAdminCredentialConfigurationInvalid()
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi NeutronAdminCredentialConfigurationInvalid: Networking client is experiencing an unauthorized exception.
2019-02-08 15:56:39.070 19147 ERROR nova.api.openstack.wsgi
2019-02-08 15:56:39.072 19147 INFO nova.api.openstack.wsgi [req-bc73b885-bab4-4116-8497-96bbe782ea4d 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.NeutronAdminCredentialConfigurationInvalid'>
2019-02-08 15:56:39.074 19147 INFO nova.osapi_compute.wsgi.server [req-bc73b885-bab4-4116-8497-96bbe782ea4d 69a638fb9e5d4788b6f23d67180f85da 2568b941b3a54c13b57c930fe3805d3c - default default] 127.0.0.1 "POST /v2.1/servers HTTP/1.0" status: 500 len: 661 time: 19.7083759

======================

And here is my nova.conf

===============
[DEFAULT]
default_availability_zone=nova
instance_usage_audit_period=hour
rootwrap_config=/etc/nova/rootwrap.conf
compute_driver=libvirt.LibvirtDriver
allow_resize_to_same_host = True
force_raw_images=True
reserved_host_memory_mb=1024
cpu_allocation_ratio=16.0
ram_allocation_ratio=1.5
instance_usage_audit=True
heal_instance_info_cache_interval=60
my_ip = 192.168.5.81
metadata_host=192.168.5.81
use_neutron=true
dhcp_domain=novalocal
firewall_driver=nova.virt.firewall.NoopFirewallDriver
state_path=/var/lib/nova
report_interval=10
service_down_time=60
enabled_apis=osapi_compute,metadata
osapi_compute_listen=127.0.0.1
osapi_compute_listen_port=8774
osapi_compute_workers=2
metadata_listen=0.0.0.0
metadata_listen_port=8775
metadata_workers=4
debug=false
log_dir=/var/log/nova
transport_url = rabbit://openstack:L3tM31n@192.168.10.11:5672/
volume_api_class=nova.volume.cinder.API
[api]
auth_strategy = keystone
[api_database]
connection = postgresql+psycopg2://packstack:L3tM31n@192.168.5.80:5432/nova_api
max_pool_size=10
[barbican]
[cache]
[cells]
[cinder]
[compute]
[conductor]
workers=4
[console]
[consoleauth]
[cors]
[database]
connection = postgresql+psycopg2://packstack:L3tM31n@192.168.5.80:5432/nova
max_pool_size = 10
[devices]
[ephemeral_storage_encryption]
[filter_scheduler]
host_subset_size=1
max_io_ops_per_host=10
max_instances_per_host=50
available_filters=nova.scheduler.filters.all_filters
enabled_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,CoreFilter
weight_classes=nova.scheduler.weights.all_weighers
[glance]
api_servers = http://127.0.0.1:9292
[guestfs]
[healthcheck]
[hyperv]
[ironic]
[key_manager]
backend=nova.keymgr.conf_key_mgr.ConfKeyManager
[keystone]
[keystone_authtoken]
www_authenticate_uri = http://127.0.0.1:7001/v3
memcached_servers = 192.168.10.11:11211,192.168.10.12:11211,192.168.10.13:11211
service_token_roles_required = True
auth_type = password
auth_url=http://127.0.0.1:7001/v3
project_domain_name = Default
user_domain_name = Default
project_name = service
username = nova
password = L3tM31n
[libvirt]
inject_password=False
inject_key=False
inject_partition=-2
cpu_mode=host-model
images_type=rbd
images_rbd_pool=volume
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = volume
rbd_secret_uuid = 7a93dd2f-be28-4f41-8d94-2aa3d1f42893
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST
[matchmaker_redis]
[metrics]
[mks]
[neutron]
ovs_bridge=br-int
default_floating_pool=public
extension_sync_interval=600
service_metadata_proxy = True
metadata_proxy_shared_secret = e56f01795df72d63c6a0
timeout=60
auth_type=password
auth_url=http://127.0.0.1:7001/v3
project_name=service
username=neutron
user_domain_name=Default
password=L3tM31n
region_name=RegionOne
endpoint_override = http://127.0.0.1:9696
[notifications]
notify_on_state_change=vm_and_task_state
[osapi_v21]
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
driver=messagingv2
[oslo_messaging_rabbit]
[oslo_messaging_zmq]
[oslo_middleware]
[oslo_policy]
[pci]
[placement]
auth_type=password
auth_url = http://127.0.0.1:7001/v3
project_name=service
project_domain_name=Default
username=placement
user_domain_name=Default
password=L3tM31n
region_name=RegionOne
[placement_database]
connection = postgresql+psycopg2://packstack:L3tM31n@192.168.5.80:5432/nova_placement
max_pool_size=10
[powervm]
[profiler]
[quota]
[rdp]
[remote_debug]
[scheduler]
driver=filter_scheduler
max_attempts=3
discover_hosts_in_cells_interval = 300
workers=4
[serial_console]
[service_user]
[spice]
enabled=true
server_listen=0.0.0.0
server_proxyclient_address= $my_ip
[upgrade_levels]
[vault]
[vendordata_dynamic_auth]
[vmware]
[vnc]
enabled=true
keymap=en-us
server_listen=0.0.0.0
server_proxyclient_address=192.168.5.81
novncproxy_base_url=http://192.168.5.81:6080/vnc_auto.html
novncproxy_host=0.0.0.0
novncproxy_port=6080
auth_schemes=none
[workarounds]
[wsgi]
api_paste_config=api-paste.ini
[xenserver]
[xvp]
[zvm]

===============

Is this a bug?
If not, kindly please tell me how to fix it.

Sincerely
-bino-

[Matt Riedemann (mriedem)]

From this config:

[neutron]
ovs_bridge=br-int
default_floating_pool=public
extension_sync_interval=600
service_metadata_proxy = True
metadata_proxy_shared_secret = e56f01795df72d63c6a0
timeout=60
auth_type=password
auth_url=http://127.0.0.1:7001/v3
project_name=service
username=neutron
user_domain_name=Default
password=L3tM31n
region_name=RegionOne
endpoint_override = http://127.0.0.1:9696

Does the 'neutron' user in the 'service' project have the 'admin' role? I think that's what it's trying to do - generate an admin role token for doing port binding to the compute host when creating the VM.

See:

https://docs.openstack.org/neutron/latest/install/controller-install-ubuntu.html#prerequisites

https://docs.openstack.org/neutron/latest/install/compute-install-ubuntu.html#configure-the-compute-service-to-use-the-networking-service

I'm not sure you need the "endpoint_override" option configured in nova.conf to talk to neutron. Nova should just use the keystone service catalog to lookup the networking API service endpoint.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]