Bug #1147994 “Auth is sending user id : tenant id instead of tok...” : Bugs : Cinder

Revision history for this message

mgmeskill@gmail.com (mgmeskill) wrote on 2013-03-06:

#1

Is there any additional info that can be supplied to help with classification or analysis of this issue?

Even some troubleshooting direction / steps would be helpful.

Thanks!
-Mike

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-12:

#2

Just to add, I have not modified my policy.json's.

Vincent Hou (houshengbo) on 2013-03-14

Changed in cinder:
status:	New → Incomplete

Revision history for this message

Vincent Hou (houshengbo) wrote on 2013-03-14:

#3

A few things need to be verified.
Is keystone successfully running? what are the other data you entered for keystone db table? e.g. endpoint...
Have you tried command like keystone --os_username=... --os_password=...--os_tenant_name=...--os_auth_url=... token-get?
What is the configuration files like for keystone and cinder?

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-14:

#4

Download full text (12.8 KiB)

Keystone is running. Instances launch fine when not doing boot from volume. I've taken instances and attached volumes to them, and the data is persistent. Just can't boot from them.

Keystone is running. Instances launch fine when not doing boot from volume. I've taken instances and attached volumes to them, and the data is persistent. Just can't boot from them.

------------------------------------------
*-/etc/keystone/Keystone.conf
------------------------------------------
# Print debugging output
verbose = True

# Print more verbose output
# (includes plaintext request logging, potentially including passwords)
debug = True

# Name of log file to output to. If not set, logging will go to stdout.
log_file = keystone.log

# The directory to keep log files in (will be prepended to --logfile)
log_dir = /var/log/keystone

# Use syslog for logging.
# use_syslog = False

# syslog facility to receive log lines
# syslog_log_facility = LOG_USER

# If this option is specified, the logging configuration file specified is
# used and overrides any other logging options specified. Please see the
# Python logging module documentation for details on logging configuration
# files.
#log_config = /etc/keystone/logging.conf

# A logging.Formatter log message format string which may use any of the
# available logging.LogRecord attributes.
# log_format = %(asctime)s %(levelname)8s [%(name)s] %(message)s

# Format string for %(asctime)s in log records.
# log_date_format = %Y-%m-%d %H:%M:%S

# onready allows you to send a notification when the process is ready to serve
# For example, to have it notify using systemd, one could set shell command:
# onready = systemd-notify --ready
# or a module with notify() method:
# onready = keystone.common.systemd

[sql]
# The SQLAlchemy connection string used to connect to the database
connection = mysql://keystone:PASSWORD@localhost:3306/keystone

# the timeout before idle sql connections are reaped
idle_timeout = 200

[identity]
driver = keystone.identity.backends.sql.Identity

[catalog]
# dynamic, sql-based backend (supports API/CLI-based management commands)
driver = keystone.catalog.backends.sql.Catalog

# static, file-based backend (does *NOT* support any management commands)
# driver = keystone.catalog.backends.templated.TemplatedCatalog

# template_file = default_catalog.templates

[token]
driver = keystone.token.backends.sql.Token

# Amount of time a token should remain valid (in seconds)
expiration = 86400

[policy]
driver = keystone.policy.backends.rules.Policy

[ec2]
driver = keystone.contrib.ec2.backends.sql.Ec2
[ssl]
#enable = True
#certfile = /etc/keystone/ssl/certs/keystone.pem
#keyfile = /etc/keystone/ssl/private/keystonekey.pem
#ca_certs = /etc/keystone/ssl/certs/ca.pem
#cert_required = True

[signing]
token_format = UUID
#certfile = /etc/keystone/ssl/certs/signing_cert.pem
#keyfile = /etc/keystone/ssl/private/signing_key.pem
#ca_certs = /etc/keystone/ssl/certs/ca.pem
#key_size = 1024
#valid_days = 3650
#ca_password = None
#token_format = PKI

[ldap]
# url = ldap://localhost
# user = dc=Manager,dc=example,dc=com
# password = None
# suffix = cn=example,cn=com
# use_dumb_member = False

# user_tree_dn = ou=Users,dc=example,dc=com
# user_objectclass = inetOrgPerson
# user_id_attribute = cn
# user_name_attribute = sn

# tenant_tree_dn = ou=Groups,dc=example,dc=com
# tenant_objectclass = groupOfNames
# tenant_id_attribute = cn
# tenant_member_attribute = member
# tenant_name_attribute = ou

# role_tree_dn = ou=Roles,dc=example,dc=com
# role_objectclass = organizationalRole
# role_id_attribute = cn
# role_member_attribute = roleOccupant

[filter:debug]
paste.filter_factory = keystone.common.wsgi:Debug.factory

[filter:token_auth]
paste.filter_factory = keystone.contrib.s3:S3Extension.factory

[filter:url_normalize]
paste.filter_factory = keystone.middleware:NormalizingFilter.factory

[filter:stats_monitoring]
paste.filter_factory = keystone.contrib.stats:StatsMiddleware.factory

[filter:stats_reporting]
paste.filter_factory = keystone.contrib.stats:StatsExtension.factory

[app:public_service]
paste.app_factory = keystone.service:public_app_factory

[app:admin_service]
paste.app_factory = keystone.service:admin_app_factory

[pipeline:public_api]
pipeline = stats_monitoring url_normalize token_auth admin_token_auth xml_body json_body debug ec2_extension user_crud_extension public_service

[pipeline:admin_api]
pipeline = stats_monitoring url_normalize token_auth admin_token_auth xml_body json_body debug stats_reporting ec2_extension s3_extension crud_extension admin_service

[app:public_version_service]
paste.app_factory = keystone.service:public_version_app_factory

[app:admin_version_service]
paste.app_factory = keystone.service:admin_version_app_factory

[pipeline:public_version_api]
pipeline = stats_monitoring url_normalize xml_body public_version_service

[pipeline:admin_version_api]
pipeline = stats_monitoring url_normalize xml_body admin_version_service

[composite:main]
use = egg:Paste#urlmap
/v2.0 = public_api
/ = public_version_api

[composite:admin]
use = egg:Paste#urlmap
/v2.0 = admin_api
/ = admin_version_api

----------------------------------                                                                                                            
*-/etc/cinder/api-paste.ini
----------------------------------
[composite:openstack_volume_api_v1]
use = call:cinder.api.auth:pipeline_factory
noauth = faultwrap sizelimit noauth osapi_volume_app_v1
keystone = faultwrap sizelimit authtoken keystonecontext osapi_volume_app_v1
keystone_nolimit = faultwrap sizelimit authtoken keystonecontext osapi_volume_app_v1

[filter:faultwrap]
paste.filter_factory = cinder.api.openstack:FaultWrapper.factory

[filter:noauth]
paste.filter_factory = cinder.api.openstack.auth:NoAuthMiddleware.factory

[filter:sizelimit]
paste.filter_factory = cinder.api.sizelimit:RequestBodySizeLimiter.factory

[app:osapi_volume_app_v1]
paste.app_factory = cinder.api.openstack.volume:APIRouter.factory

[pipeline:osvolumeversions]
pipeline = faultwrap osvolumeversionapp

[app:osvolumeversionapp]
paste.app_factory = cinder.api.openstack.volume.versions:Versions.factory

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = cinder.api.auth:CinderKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystone.middleware.auth_token:filter_factory
service_protocol = http
service_host = 10.101.54.27
service_port = 5000
auth_host = 10.101.54.27
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = cinder
admin_password = PASSWORD
#admin_token = supersecrettoken
--------------------------------------------

-----------------------------------                                                  
*-/etc/cinder/cinder.conf
-----------------------------------
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
sql_connection = mysql://cinder:PASSWORD@10.101.54.27/cinder
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
debug = True
auth_strategy=keystone
state_path = /var/lib/cinder
volumes_dir = /var/lib/cinder/volumes

glance_host=10.101.54.27
glance_port=9292
glance_api_servers=10.101.54.27:9292
glance_api_version=1
glance_num_retries=3
glance_api_insecure=true

### Enable Ceph storage for Volumes
volume_driver = cinder.volume.driver.RBDDriver
rbd_pool = volumes
rbd_user = volumes

### Enable quotas
quota_volumes = 1000

### RabbitMQ Stuff
rabbit_host = 10.101.54.27
rabbit_port = 5672
rabbit_userid = guest
rabbit_password = guest
rabbit_virtual_host = /
----------------------------

When specify OS Username and password I still get the same output in cinder's log.

--------------------------------------
nova --debug --os-username=admin --os-password=PASSWORD --os-auth-url=http://10.101.54.27:5000/v2.0  boot --image f91c7c9e-31b5-4401-b790-8ddf34f2cc2a --flavor 3 --block-device-mapping vda=c163063a-32b4-4173-ad57-92056eed35c1:::0 Linux-bfv-zb11-2
--------------------------------------

--------------------------------------
tail -n 100 /var/log/cinder/cinder-api.log
--------------------------------------
2013-03-14 11:01:22 AUDIT cinder.api.openstack.volume.volumes [req-21881520-050e-4bc3-88ec-97c5320b1e2e 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] vol={'volume_metadata': [], 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2013, 3, 6, 18, 57, 47), 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'c163063a-32b4-4173-ad57-92056eed35c1', 'size': 10L, 'user_id': u'07aedb39c8e0472aad8f4a2024b87205', 'attach_time': None, 'display_description': None, 'project_id': u'ac37039f5fba4874961fdacf672f2fef', 'launched_at': datetime.datetime(2013, 3, 6, 18, 57, 47), 'scheduled_at': datetime.datetime(2013, 3, 6, 18, 57, 46), 'status': u'available', 'volume_type_id': None, 'deleted': False, 'provider_location': None, 'host': u'atmos27', 'provider_auth': None, 'display_name': u'Cirros_64-bfv', 'instance_uuid': None, 'created_at': datetime.datetime(2013, 3, 6, 18, 57, 46), 'attach_status': u'detached', 'volume_type': None}
2013-03-14 11:01:22 INFO cinder.api.openstack.wsgi [req-21881520-050e-4bc3-88ec-97c5320b1e2e 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] http://10.101.54.27:8776/v1/ac37039f5fba4874961fdacf672f2fef/volumes/c163063a-32b4-4173-ad57-92056eed35c1 returned with HTTP 200
2013-03-14 11:01:24 6917 WARNING keystone.middleware.auth_token [-] Authorization failed for token 07aedb39c8e0472aad8f4a2024b87205:ac37039f5fba4874961fdacf672f2fef
2013-03-14 11:01:24 6917 WARNING keystone.middleware.auth_token [-] Authorization failed for token 07aedb39c8e0472aad8f4a2024b87205:ac37039f5fba4874961fdacf672f2fef
2013-03-14 11:01:24 6917 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request

Revision history for this message

Vincent Hou (houshengbo) wrote on 2013-03-15:

#5

Hi bullardza, I am testing booting from the volume with the latest version of OpenStack, and file the bug
https://bugs.launchpad.net/cinder/+bug/1155512

For F version, copying image to volume and vice versa have not been added yet, so you need to attach an empty volume to a VM, access the VM, download the image, copy the image into the volume by commands, detach the volume and in the end boot from this volume. Is it how you did it? Please refer to
http://docs.openstack.org/folsom/openstack-compute/admin/content/boot-from-volume.html

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-15:

#6

Yes. With the CIrros image and a Windows image we created. We've actually gotten it working with our images before, but ran into some other issues that caused us to rebuild the test environment with the latest release. I will go through the volume setup again to be sure I am following it to a T.

Aside from that, what would explain the token behavior? Regardless of the underlying volumes boot-ability, should it be sending User ID : Tenant ID instead of a real token? On other requests, the Cinder-api.log shows a real token...........nova boot and from horizon without volumes and no token problems......the deploy just fine.

Revision history for this message

Vincent Hou (houshengbo) wrote on 2013-03-18:

#7

For your environment, I think it is enough to set the variables:
export OS_TENANT_NAME=demo
export OS_USERNAME=admin
export OS_PASSWORD=PASSWORD
export OS_AUTH_URL="http://10.101.54.27:35357/v2.0"
Make sure they are correct.

A auth_token needs to be set in the field "X-Auth-Token" of the header for each http request. However, it you do not have the token, a token can be generated for you if you provide your OS_USERNAME and OS_PASSWORD, and then set it into X-Auth-Token". OS_TENANT_NAME is optional. The code of client side will do the authentication automatically if you d not provide the token but provide username and password.

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-19:

#8

Download full text (5.0 KiB)

I commented out everything except what you listed for my rc file that I am sourcing. Didn't seem to make a difference on my cinder problem.

I tried some token get and am able to get them fine.
_____________________________________________________
root@atmos27:~# keystone --debug --os-username=admin --os-password=PASSWORD --os-auth-url=http://10.101.54.27:35357/v2.0 token-get

connect: (10.101.54.27, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 10.101.54.27:35357\r\nContent-Length: 80\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-keystoneclient\r\n\r\n{"auth": {"passwordCredentials": {"username": "admin", "password": "PASSWORD"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Content-Length: 244
header: Date: Tue, 19 Mar 2013 14:17:25 GMT
No handlers could be found for logger "keystoneclient.v2_0.client"
+----------+----------------------------------+
| Property | Value |
+----------+----------------------------------+
| expires | 2013-03-20T14:17:25Z |
| id | b98af5515d194708907385a01c4089f3 |
| user_id | 07aedb39c8e0472aad8f4a2024b87205 |

______________________________________________________

So getting a token is working. So doing it with the cinder user......
______________________________________________________
root@atmos27:~# keystone --debug --os-username=cinder --os-password=PASSWORD --os-tenant-name=service --os-auth-url=http://10.101.54.27:35357/v2.0 token-get

connect: (10.101.54.27, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 10.101.54.27:35357\r\nContent-Length: 106\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-keystoneclient\r\n\r\n{"auth": {"tenantName": "service", "passwordCredentials": {"username": "cinder", "password": "PASSWORD"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Content-Length: 2706
header: Date: Tue, 19 Mar 2013 14:19:21 GMT
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2013-03-20T14:19:21Z |
| id | cb607751951943d080a500cb1fef958e |
| tenant_id | 001c735a4c5640b389a51fc51d1626f9 |
| user_id | 98be9a494c1b41118a664b8b58e4af2f |
+-----------+----------------------------------+

__________________________________________________________

Token get with Admin in the demo tenant works. Works with Cinder user in Service tenant as well. So I am pretty sure my token get is working. Just want to get past this token part of booting from volumes.....when I nova boot with boot from volume, it fails and cinder-api logs this:
______________________________________________________

2013-03-19 09:09:33 6917 WARNING keystone.middleware.auth_token [-] Authorization failed for token 07aedb39c8e0472aad8f4a2024b87205:ac37039f5fba4874961fdacf672f2fef
2013-03-19 09:09:33 6917 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request
______________________________________________________

The first part of the faile...

I commented out everything except what you listed for my rc file that I am sourcing. Didn't seem to make a difference on my cinder problem.

I tried some token get and am able to get them fine.
_____________________________________________________
root@atmos27:~# keystone --debug --os-username=admin --os-password=PASSWORD --os-auth-url=http://10.101.54.27:35357/v2.0 token-get

connect: (10.101.54.27, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 10.101.54.27:35357\r\nContent-Length: 80\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-keystoneclient\r\n\r\n{"auth": {"passwordCredentials": {"username": "admin", "password": "PASSWORD"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Content-Length: 244
header: Date: Tue, 19 Mar 2013 14:17:25 GMT
No handlers could be found for logger "keystoneclient.v2_0.client"
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
| expires  |       2013-03-20T14:17:25Z       |
|    id    | b98af5515d194708907385a01c4089f3 |
| user_id  | 07aedb39c8e0472aad8f4a2024b87205 |

______________________________________________________

So getting a token is working. So doing it with the cinder user......
______________________________________________________
root@atmos27:~#  keystone --debug --os-username=cinder --os-password=PASSWORD --os-tenant-name=service --os-auth-url=http://10.101.54.27:35357/v2.0 token-get

connect: (10.101.54.27, 35357)
send: 'POST /v2.0/tokens HTTP/1.1\r\nHost: 10.101.54.27:35357\r\nContent-Length: 106\r\ncontent-type: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: python-keystoneclient\r\n\r\n{"auth": {"tenantName": "service", "passwordCredentials": {"username": "cinder", "password": "PASSWORD"}}}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Vary: X-Auth-Token
header: Content-Type: application/json
header: Content-Length: 2706
header: Date: Tue, 19 Mar 2013 14:19:21 GMT
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2013-03-20T14:19:21Z       |
|     id    | cb607751951943d080a500cb1fef958e |
| tenant_id | 001c735a4c5640b389a51fc51d1626f9 |
|  user_id  | 98be9a494c1b41118a664b8b58e4af2f |
+-----------+----------------------------------+

__________________________________________________________

Token get with Admin in the demo tenant works. Works with Cinder user in Service tenant as well. So I am pretty sure my token get is working. Just want to get past this token part of booting from volumes.....when I nova boot with boot from volume, it fails and cinder-api logs this:
______________________________________________________

2013-03-19 09:09:33 6917 WARNING keystone.middleware.auth_token [-] Authorization failed for token 07aedb39c8e0472aad8f4a2024b87205:ac37039f5fba4874961fdacf672f2fef
2013-03-19 09:09:33 6917 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request
______________________________________________________

The second part "ac37039f5fba4874961fdacf672f2fef" is the tenant demo:
__________________________________________________________

Specifying the tenant and user id in my nova boot command doesnt seem to matter. The instance shows up really quickly but in an errored state, and does not delete cleanly.

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-20:

#9

Download full text (8.3 KiB)

I found this today when experimenting with boot from volume vs regular instances....I tailed /var/log/nova/nova-compute.log during a nova boot..............Instance failed block device setup who's timestamp is the same as the cinder-api.logs message about token rejection..........

2013-03-20 15:06:25 DEBUG nova.volume.cinder [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] Cinderclient connection created using URL: http://10.101.54.27:8776/v1/ac37039f5fba4874961fdacf672f2fef cinderclient /usr/lib/python2.7/dist-packages/nova/volume/cinder.py: 68
2013-03-20 15:06:25 ERROR nova.compute.manager [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] [instan ce: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Instance failed block device setup
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Traceback (most recent call last):
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/nova/compute/manager .py", line 729, in _prep_block_device
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] return self._setup_block_device_mapping(context, instance )
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/nova/compute/manager .py", line 447, in _setup_block_device_mapping
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] volume = self.volume_api.get(context, bdm['volume_id'])
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.p y", line 144, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] item = cinderclient(context).volumes.get(volume_id)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volu mes.py", line 147, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] return self._get("/volumes/%s" % volume_id, "volume")
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/cinderclient/base.py ", line 141, in _get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] resp, body = self.api.client.get(url)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] File "/usr/lib/python2.7/dist-packages/cinderclient/client. py", line 138, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] return self._cs_request(url, 'GET', **kwargs)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2...

I found this today when experimenting with boot from volume vs regular instances....I tailed /var/log/nova/nova-compute.log during a nova boot..............Instance failed block device setup who's timestamp is the same as the cinder-api.logs message about token rejection..........

2013-03-20 15:06:25 DEBUG nova.volume.cinder [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] Cinderclient connection created using URL: http://10.101.54.27:8776/v1/ac37039f5fba4874961fdacf672f2fef cinderclient /usr/lib/python2.7/dist-packages/nova/volume/cinder.py:   68
2013-03-20 15:06:25 ERROR nova.compute.manager [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] [instan   ce: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Instance failed block device setup
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Traceback (most recent call last):
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager   .py", line 729, in _prep_block_device
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     return self._setup_block_device_mapping(context, instance   )
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager   .py", line 447, in _setup_block_device_mapping
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     volume = self.volume_api.get(context, bdm['volume_id'])
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.p   y", line 144, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     item = cinderclient(context).volumes.get(volume_id)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volu   mes.py", line 147, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     return self._get("/volumes/%s" % volume_id, "volume")
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/base.py   ", line 141, in _get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     resp, body = self.api.client.get(url)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/client.   py", line 138, in get
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     return self._cs_request(url, 'GET', **kwargs)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/client.   py", line 130, in _cs_request
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     self.authenticate()
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/client.   py", line 246, in authenticate
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     auth_url = self._v1_auth(auth_url)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/client.   py", line 264, in _v1_auth
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     resp, body = self.request(url, 'GET', headers=headers)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]   File "/usr/lib/python2.7/dist-packages/cinderclient/client.   py", line 109, in request
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]     raise exceptions.from_response(resp, body)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] BadRequest: n/a (HTTP 400)
2013-03-20 15:06:25 14470 TRACE nova.compute.manager [instance: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13]
2013-03-20 15:06:25 DEBUG nova.utils [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] Got semaphore "co   mpute_resources" for method "abort_resource_claim"... inner /usr/lib/python2.7/dist-packages/nova/utils.py:765
2013-03-20 15:06:25 INFO nova.compute.resource_tracker [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef]    Aborting claim: [Claim 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13: 4096 MB memory, 40 GB disk, 2 VCPUS]
2013-03-20 15:06:25 DEBUG nova.compute.manager [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] [instan   ce: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Deallocating network for instance _deallocate_network /usr/lib/python2.7/dist-packages/nova/compute/manager.py:773
2013-03-20 15:06:25 DEBUG nova.network.quantumv2.api [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] d   eallocate_for_instance() for Linux-bfvzb-3202013-1 deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py:171
2013-03-20 15:06:26 DEBUG nova.compute.manager [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] [instan   ce: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Re-scheduling instance: attempt 1 _reschedule /usr/lib/python2.7/dist-packages/nova/compute/manager.py:579
2013-03-20 15:06:26 DEBUG nova.utils [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] Got semaphore "co   mpute_resources" for method "update_usage"... inner /usr/lib/python2.7/dist-packages/nova/utils.py:765
2013-03-20 15:06:26 14470 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on scheduler... cast /usr/lib/python2.7/dist-packages/nova/openstack/co   mmon/rpc/amqp.py:377
2013-03-20 15:06:26 ERROR nova.compute.manager [req-a49f2cfc-6e35-498d-9b90-60df72b19902 07aedb39c8e0472aad8f4a2024b87205 ac37039f5fba4874961fdacf672f2fef] [instan   ce: 6d1e115b-6201-4ba4-a088-2e0fe3a4ca13] Build error: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py",    line 491, in _run_instance\n    instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 729, in _prep_block_device\n    return self   ._setup_block_device_mapping(context, instance)\n', '  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 447, in _setup_block_device_mapping\n       volume = self.volume_api.get(context, bdm[\'volume_id\'])\n', '  File "/usr/lib/python2.7/dist-packages/nova/volume/cinder.py", line 144, in get\n    item = cin   derclient(context).volumes.get(volume_id)\n', '  File "/usr/lib/python2.7/dist-packages/cinderclient/v1/volumes.py", line 147, in get\n    return self._get("/volum   es/%s" % volume_id, "volume")\n', '  File "/usr/lib/python2.7/dist-packages/cinderclient/base.py", line 141, in _get\n    resp, body = self.api.client.get(url)\n',    '  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 138, in get\n    return self._cs_request(url, \'GET\', **kwargs)\n', '  File "/usr/lib/pyt   hon2.7/dist-packages/cinderclient/client.py", line 130, in _cs_request\n    self.authenticate()\n', '  File "/usr/lib/python2.7/dist-packages/cinderclient/client.p   y", line 246, in authenticate\n    auth_url = self._v1_auth(auth_url)\n', '  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 264, in _v1_auth\   n    resp, body = self.request(url, \'GET\', headers=headers)\n', '  File "/usr/lib/python2.7/dist-packages/cinderclient/client.py", line 109, in request\n    rais   e exceptions.from_response(resp, body)\n', 'BadRequest: n/a (HTTP 400)\n']

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-03-20:

#10

Why wouldn't I also get the same token rejection when creating a volume?

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-04-02:

#11

Is anybody else having this occur when doing boot from volume with Ceph on Openstack Folsom?

Revision history for this message

bullardza@gmail.com (bullardza) wrote on 2013-04-16:

#12

Vincent, Is there any more output I can show you to get rolling on troubleshooting this?

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2015-10-31: Cleanup

#13

Closing stale bug. If this is still an issue please reopen.

Changed in cinder:
status:	Incomplete → Invalid

Cinder

Auth is sending user id : tenant id instead of token to cinder API

Bug Description

Other bug subscribers

Remote bug watches