Bug #1481255 ""nova service-list” is not working but “nova endpo...” : Bugs : openstack-manuals

Revision history for this message

Christian Berendt (berendt) wrote on 2015-08-04:

#1

Have you sourced the admin-openrc.sh file prior to running nova service-list?

Have you checked the nova logfile (/var/log/nova) and the keystone logfile (/var/log/keystone)?

Have you checked the nova configuration file (/etc/nova/nova.conf) for the correct credentials?

Normally this error means that the nova service cannot authenticate against the keystone service.

Changed in openstack-manuals:
status:	New → Incomplete

Revision history for this message

Udayendu Kar (udayendu-kar) wrote on 2015-08-05:

#2

Download full text (15.0 KiB)

Hello berendt,

Thanks for looking into it.

Please find the reply to your below query:

>>> Have you sourced the admin-openrc.sh file prior to running nova service-list?

Yes, I did but still have the same issue. Here is some out put for your reference:

Hello berendt,

Thanks for looking into it.

Please find the reply to your below query:

>>> Have you sourced the admin-openrc.sh file prior to running nova service-list?

Yes, I did but still have the same issue.  Here is some out put for your reference:

>>> Have you checked the nova logfile (/var/log/nova)  ?

Yes, its showing the authentication issue. Please find the below logs. I followed the docs and did this configuration.
----
2015-08-05 08:03:25.165 1857 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2015-08-05 08:03:25.238 1857 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2015-08-05 08:03:25.238 1857 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-08-05 08:03:25.240 1857 INFO nova.osapi_compute.wsgi.server [-] 192.168.122.150 "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" status: 401 len: 280 time: 0.1506732
2015-08-05 08:03:25.396 1857 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2015-08-05 08:03:25.467 1857 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}
2015-08-05 08:03:25.468 1857 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-08-05 08:03:25.472 1857 INFO nova.osapi_compute.wsgi.server [-] 192.168.122.150 "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" status: 401 len: 280 time: 0.1449728
----

>>> the keystone logfile (/var/log/keystone) ?

Keystone is also showing the authentication issue:

----
2015-08-05 08:05:56.567 1994 INFO keystone.common.wsgi [-] GET /?
2015-08-05 08:05:56.572 1993 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:56.652 1992 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:56.773 1992 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.122.150
2015-08-05 08:05:56.782 1995 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:56.857 1995 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.122.150
2015-08-05 08:05:56.866 1991 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:56.951 1994 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:57.010 1994 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.122.150
2015-08-05 08:05:57.016 1993 INFO keystone.common.wsgi [-] POST /auth/tokens?
2015-08-05 08:05:57.076 1993 WARNING keystone.common.wsgi [-] Authorization failed. The request you have made requires authentication. from 192.168.122.150
----

>>>  Have you checked the nova configuration file (/etc/nova/nova.conf) for the correct credentials?

credentials seems to be correct. Still I am sending you the complete config from both controller and compute. Hope you will find something in it:

---- from controller ----
# cat /etc/nova/nova.conf 
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=ec2,osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 192.168.122.150
vncserver_listen = 192.168.122.150
vncserver_proxyclient_address = 192.168.122.150

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[glance]
host = controller

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = xxxx

[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = xxxx

[database]
connection = mysql://nova:xxxx@controller/nova

---- from compute ----
# cat /etc/nova/nova.conf 
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=ec2,osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 192.168.122.151
vnc_enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 192.168.122.151
novncproxy_base_url = http://192.168.122.150:6080/vnc_auto.html

[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = xxxx

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = xxxx

[glance]
host = controller

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

>>> Normally this error means that the nova service cannot authenticate against the keystone service.

Yes, absolutely correct. I believe some issue is there is the api version that we are using here. For example, if I want to use keystone command to list services or users or roles then using same "admin-openrc.sh" I should be able to do it but thats not working. So for key stone to work I have to use it separately with api v2.0 . Here is the example for you:

# cat admin-openrc.sh 
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=xxxx
export OS_AUTH_URL=http://controller:35357/v3     <======
export OS_IMAGE_API_VERSION=2
root@controller:~# source admin-openrc.sh 
root@controller:~# keystone service-list
Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-5ccbb2e6-1d11-400d-865d-d9d4ec15bc1f)
root@controller:~# keystone user-list
Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-b089bed1-e517-4d5f-9b60-194164d5516f)

The above strange behaviour forced me to think that might be some issue is there in the API version itself which is causing this issue and if so then it needs a fix.

Here is the keystone's endpoint list:

root@controller:~# keystone endpoint-list
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+
|                id                |   region  |                publicurl                |               internalurl               |                 adminurl                |            service_id            |
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+
| 05faa47ed2e54ed09ab762e93b6454ee | RegionOne |       http://controller:5000/v2.0       |       http://controller:5000/v2.0       |       http://controller:35357/v2.0      | e2c81fff8d0f43a781ed3a35a88e1be3 |
| a1a9861ab14849b0bb271f77713ed2ce | RegionOne | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | 241f31a69a8e4255a37fc2dba850b997 |
| f185c6ff2190442dbe0be8cc93cd4ef0 | RegionOne |          http://controller:9292         |          http://controller:9292         |          http://controller:9292         | b8f7ae274bac47eca2ed5b36981fa5ea |
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+

do you think anything wrong is there with the adminurl for  keystone service ??

Waiting for your response.

Thanks,
Uday

Revision history for this message

Udayendu Kar (udayendu-kar) wrote on 2015-08-05:

#3

Download full text (9.6 KiB)

Here are some info with debug option for your reference:

# nova --debug service-list
DEBUG (session:195) REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:188) Starting new HTTP connection (1): controller
DEBUG (connectionpool:362) "GET /v3 HTTP/1.1" 200 250
DEBUG (session:223) RESP: [200] content-length: 250 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Wed, 05 Aug 2015 03:00:35 GMT x-openstack-request-id: req-b434334e-332d-424c-b94b-0afbe8844c38 content-type: application/json x-distribution: Ubuntu
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://controller:35357/v3/", "rel": "self"}]}}

DEBUG (v3:125) Making authentication request to http://controller:35357/v3/auth/tokens
DEBUG (connectionpool:362) "POST /v3/auth/tokens HTTP/1.1" 201 2257
DEBUG (iso8601:184) Parsed 2015-08-05T04:00:36.039888Z into {'tz_sign': None, 'second_fraction': u'039888', 'hour': u'04', 'daydash': u'05', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'36', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'08', 'day': None, 'minute': u'00'} with default timezone <iso8601.iso8601.Utc object at 0x7f56cf813a10>
DEBUG (iso8601:140) Got u'2015' for 'year' with default None
DEBUG (iso8601:140) Got u'08' for 'monthdash' with default 1
DEBUG (iso8601:140) Got 8 for 'month' with default 8
DEBUG (iso8601:140) Got u'05' for 'daydash' with default 1
DEBUG (iso8601:140) Got 5 for 'day' with default 5
DEBUG (iso8601:140) Got u'04' for 'hour' with default None
DEBUG (iso8601:140) Got u'00' for 'minute' with default None
DEBUG (iso8601:140) Got u'36' for 'second' with default None
DEBUG (session:195) REQ: curl -g -i -X GET http://controller:8774/v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}534a02fc026b183a3f06d272e3b9e55bfd011096"
INFO (connectionpool:188) Starting new HTTP connection (1): controller
DEBUG (connectionpool:362) "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" 401 23
DEBUG (session:223) RESP:
DEBUG (v3:125) Making authentication request to http://controller:35357/v3/auth/tokens
DEBUG (connectionpool:362) "POST /v3/auth/tokens HTTP/1.1" 201 2257
DEBUG (connectionpool:362) "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" 401 23
DEBUG (session:223) RESP:
DEBUG (shell:914) Unauthorized (HTTP 401) (Request-ID: req-b86bcfd5-7a2b-4c10-9407-f1e94293d67b)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 911, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 838, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 3442, in do_service_list
    result = cs.services.list(host=args.host, binary=args.binary)
  File "/usr/lib/python2.7/dis...

Here are some info with debug option for your reference:

# nova --debug service-list
DEBUG (session:195) REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:188) Starting new HTTP connection (1): controller
DEBUG (connectionpool:362) "GET /v3 HTTP/1.1" 200 250
DEBUG (session:223) RESP: [200] content-length: 250 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Wed, 05 Aug 2015 03:00:35 GMT x-openstack-request-id: req-b434334e-332d-424c-b94b-0afbe8844c38 content-type: application/json x-distribution: Ubuntu 
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://controller:35357/v3/", "rel": "self"}]}}

DEBUG (v3:125) Making authentication request to http://controller:35357/v3/auth/tokens
DEBUG (connectionpool:362) "POST /v3/auth/tokens HTTP/1.1" 201 2257
DEBUG (iso8601:184) Parsed 2015-08-05T04:00:36.039888Z into {'tz_sign': None, 'second_fraction': u'039888', 'hour': u'04', 'daydash': u'05', 'tz_hour': None, 'month': None, 'timezone': u'Z', 'second': u'36', 'tz_minute': None, 'year': u'2015', 'separator': u'T', 'monthdash': u'08', 'day': None, 'minute': u'00'} with default timezone <iso8601.iso8601.Utc object at 0x7f56cf813a10>
DEBUG (iso8601:140) Got u'2015' for 'year' with default None
DEBUG (iso8601:140) Got u'08' for 'monthdash' with default 1
DEBUG (iso8601:140) Got 8 for 'month' with default 8
DEBUG (iso8601:140) Got u'05' for 'daydash' with default 1
DEBUG (iso8601:140) Got 5 for 'day' with default 5
DEBUG (iso8601:140) Got u'04' for 'hour' with default None
DEBUG (iso8601:140) Got u'00' for 'minute' with default None
DEBUG (iso8601:140) Got u'36' for 'second' with default None
DEBUG (session:195) REQ: curl -g -i -X GET http://controller:8774/v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}534a02fc026b183a3f06d272e3b9e55bfd011096"
INFO (connectionpool:188) Starting new HTTP connection (1): controller
DEBUG (connectionpool:362) "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" 401 23
DEBUG (session:223) RESP:
DEBUG (v3:125) Making authentication request to http://controller:35357/v3/auth/tokens
DEBUG (connectionpool:362) "POST /v3/auth/tokens HTTP/1.1" 201 2257
DEBUG (connectionpool:362) "GET /v2/e7491bb2026e4b12bdfb938820f2c3c6/os-services HTTP/1.1" 401 23
DEBUG (session:223) RESP:
DEBUG (shell:914) Unauthorized (HTTP 401) (Request-ID: req-b86bcfd5-7a2b-4c10-9407-f1e94293d67b)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 911, in main
    OpenStackComputeShell().main(argv)
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 838, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/shell.py", line 3442, in do_service_list
    result = cs.services.list(host=args.host, binary=args.binary)
  File "/usr/lib/python2.7/dist-packages/novaclient/v2/services.py", line 49, in list
    return self._list(url, "services")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/adapter.py", line 164, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 96, in request
    raise exceptions.from_response(resp, body, url, method)
Unauthorized: Unauthorized (HTTP 401) (Request-ID: req-b86bcfd5-7a2b-4c10-9407-f1e94293d67b)
ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-b86bcfd5-7a2b-4c10-9407-f1e94293d67b)

root@controller:~# nova --debug endpoints
DEBUG (session:195) REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:188) Starting new HTTP connection (1): controller
DEBUG (connectionpool:362) "GET /v3 HTTP/1.1" 200 250
DEBUG (session:223) RESP: [200] content-length: 250 vary: X-Auth-Token keep-alive: timeout=5, max=100 server: Apache/2.4.7 (Ubuntu) connection: Keep-Alive date: Wed, 05 Aug 2015 03:01:33 GMT x-openstack-request-id: req-46d1d02c-c69c-4853-991b-3be33f6d3bfb content-type: application/json x-distribution: Ubuntu 
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://controller:35357/v3/", "rel": "self"}]}}

Udayendu Kar (udayendu-kar) on 2015-08-05

Changed in openstack-manuals:
status:	Incomplete → New

Revision history for this message

Udayendu Kar (udayendu-kar) wrote on 2015-08-05:

#4

Some additional info. And to collect these I have sourced the below:

# cat keystone-openrc.sh
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=xxxx
export OS_AUTH_URL=http://controller:35357/v2.0 <===== [With V3 its not working]
export OS_IMAGE_API_VERSION=2

root@controller:~# keystone user-role-list
+----------------------------------+-------+----------------------------------+----------------------------------+
| id | name | user_id | tenant_id |
+----------------------------------+-------+----------------------------------+----------------------------------+
| 65d34f32a1384f9283e612bad3179bc7 | admin | 1f44ac8ee04b4e8c9d49a605c8707b76 | e7491bb2026e4b12bdfb938820f2c3c6 |
+----------------------------------+-------+----------------------------------+----------------------------------+

Some additional info. And to collect these I have sourced the below:

# cat keystone-openrc.sh 
export OS_PROJECT_DOMAIN_ID=default
export OS_USER_DOMAIN_ID=default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=xxxx
export OS_AUTH_URL=http://controller:35357/v2.0   <===== [With V3 its not working]
export OS_IMAGE_API_VERSION=2

root@controller:~# keystone user-role-list
+----------------------------------+-------+----------------------------------+----------------------------------+
|                id                |  name |             user_id              |            tenant_id             |
+----------------------------------+-------+----------------------------------+----------------------------------+
| 65d34f32a1384f9283e612bad3179bc7 | admin | 1f44ac8ee04b4e8c9d49a605c8707b76 | e7491bb2026e4b12bdfb938820f2c3c6 |
+----------------------------------+-------+----------------------------------+----------------------------------+

Revision history for this message

Andreas Jaeger (jaegerandi) wrote on 2015-08-20:

#5

We cannot help debug, please ask on openstack mailing list or use ask.openstack.org

Changed in openstack-manuals:
status:	New → Invalid

openstack-manuals

"nova service-list" is not working but "nova endpoints" is working fine

Bug Description

Other bug subscribers

Remote bug watches