cinder list - ERROR: Not found

[Federico Iezzi]

Hello,

In my OpenStack Grizzly setup I have a Ceph cluster as backend for Cinder.
I'm sure that configuration of cinder.conf and libvirt to talk with Ceph is right (after start of service Cinder talk with Ceph asking "rados lspools" and it respond).

BTW with cinder client and nova client, I'm not able to talk with cinder backend.

# cinder list

==> /var/log/cinder/cinder-api.log <==
2013-03-25 20:07:14 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token
2013-03-25 20:07:14 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
2013-03-25 20:07:14 DEBUG [routes.middleware] No route matched for GET /volumes/detail
ERROR: Not found

# cinder create 10

==> /var/log/cinder/cinder-api.log <==
2013-03-25 20:08:13 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token
2013-03-25 20:08:13 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
2013-03-25 20:08:13 DEBUG [routes.middleware] No route matched for POST /volumes
ERROR: Not found

With Strace

strace -e trace=network cinder list
--- SIGCHLD (Child exited) @ 0 (0) ---
--- SIGCHLD (Child exited) @ 0 (0) ---
--- SIGCHLD (Child exited) @ 0 (0) ---
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
sendto(3, "\233\306\1\0\0\1\0\0\0\0\0\0\5swift\5enter\2it\0\0\34\0\1", 32, MSG_NOSIGNAL, NULL, 0) = 32
recvfrom(3, "\233\306\201\200\0\1\0\0\0\1\0\0\5swift\5enter\2it\0\0\34\0\1"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 85
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
sendto(3, "\344\251\1\0\0\1\0\0\0\0\0\0\5swift\5enter\2it\0\0\34\0\1", 32, MSG_NOSIGNAL, NULL, 0) = 32
recvfrom(3, "\344\251\201\200\0\1\0\0\0\1\0\0\5swift\5enter\2it\0\0\34\0\1"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 85
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, 16) = 0
sendto(3, "\327\235\1\0\0\1\0\0\0\0\0\0\5swift\5enter\2it\0\0\1\0\1", 32, MSG_NOSIGNAL, NULL, 0) = 32
recvfrom(3, "\327\235\201\200\0\1\0\1\0\0\0\0\5swift\5enter\2it\0\0\1\0\1"..., 1024, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("8.8.8.8")}, [16]) = 48
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("212.29.xxx.yyy")}, 16) = 0
sendto(3, "POST /v2.0/tokens HTTP/1.1\r\nHost"..., 320, 0, NULL, 0) = 320
recvfrom(3, "HTTP/1.1 200 OK\r\nVary: X-Auth-To"..., 8192, 0, NULL, NULL) = 1448
recvfrom(3, "In1dLCAiZW5kcG9pbnRzX2xpbmtzIjog"..., 5986, 0, NULL, NULL) = 5986
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(8776), sin_addr=inet_addr("212.29.xxx.yyy")}, 16) = 0
sendto(3, "GET /v2/volumes/detail HTTP/1.1\r"..., 4580, 0, NULL, 0) = 4580
recvfrom(3, 2013-03-25 20:10:42 DEBUG [keystoneclient.middleware.auth_token] Authenticating user token
2013-03-25 20:10:42 DEBUG [keystoneclient.middleware.auth_token] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role
2013-03-25 20:10:42 DEBUG [routes.middleware] No route matched for GET /volumes/detail
"HTTP/1.1 404 Not Found\r\nContent-"..., 8192, 0, NULL, NULL) = 195
ERROR: Not found

The config of policy.json is the default.
Below the cinder.conf

[DEFAULT]
debug=true
verbose=true
log_date_format=%Y-%m-%d %H:%M:%S
connection_type=libvirt
sql_connection=mysql://cinder:cinder@10.1.2.50/cinder
api_paste_config=/etc/cinder/api-paste.ini
pybasedir=/usr/lib/python2.7/dist-packages
bindir=$pybasedir/bin
my_ip=10.1.2.51
glance_host=10.1.2.52
glance_port=9292
glance_api_servers=$glance_host:$glance_port
glance_num_retries=0
glance_api_insecure=true
scheduler_topic=cinder-scheduler
volume_topic=cinder-volume
enable_v2_api=true
api_rate_limit=true
osapi_volume_extension=cinder.api.contrib.standard_extensions
osapi_max_limit=1000
sql_max_retries=10
host=nova1
storage_availability_zone=vega
auth_strategy=keystone
control_exchange=cinder
policy_file=policy.json
policy_default_rule=default
volume_name_template=volume-%s
snapshot_name_template=snapshot-%s
backup_name_template=backup-%s
rabbit_host=10.1.2.50
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=nova-pool
rbd_user=openstack
rbd_secret_uuid=52f0fa68-87cd-680d-2850-888d981c4896

Could anyone explane me if this is an bug or my config error?

Thanks you so much!
Federico

[Mike Perez (thingee)]

Hi Federico,

The API server and client look like they're configured correctly. However I noticed when the API server receives the forwarded request it comes through as /volumes/detail without a version.

When you run "keystone service-list shows" what's the endpoint for Cinder look like?

[Juan Pavlik (jjpavlik)]

Hi Mike, i'm going through the same problem.

root@heladera:/etc/cinder# cinder --os-username XXX --os-tenant-name XXX --os-password XXX list
ERROR: [Errno 113] No route to host
root@heladera:/etc/cinder#

I run keystone in a separate node:

root@asado:~# keystone service-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+----------+--------------+------------------------------+
| id | name | type | description |
+----------------------------------+----------+--------------+------------------------------+
| a51535b7482a4f9580fb5fdc44c24b93 | cinder | volume | OpenStack Volume Service |
| e8b017f0ce944df0b2a1785df1a3c3c6 | cinder | volume | OpenStack Volume Service |
| 614d7b40ceb14722b5a804d81d683fa7 | ec2 | ec2 | OpenStack EC2 service |
| df49235972dc491182ef698ae3d89174 | ec2 | ec2 | OpenStack EC2 service |
| 24ab4d927d6046eea58c6b72812bc32e | glance | image | OpenStack Image Service |
| d8255833fd1544618e608891ab9a4728 | glance | image | OpenStack Image Service |
| 7e077898e7034068b70b772f5d25f531 | keystone | identity | OpenStack Identity |
| e742806a7c9f40f1bc97e0353b9e9d81 | keystone | identity | OpenStack Identity |
| 70d9cbc074564420a0002b67968433e9 | nova | compute | OpenStack Compute Service |
| f28ebdec57cf48d9a1a4fb454e0384f0 | nova | compute | OpenStack Compute Service |
| 109b47bfaa3b402189f41fe950f68aa6 | quantum | network | OpenStack Networking service |
| 4950e14d92604648856c604033b2aec0 | quantum | network | OpenStack Networking service |
| 4f46241182524af385f1d9961323c846 | swift | object-store | OpenStack Storage Service |
| 9f5003cedb8e468a91353b37e78f26d0 | swift | object-store | OpenStack Storage Service |
+----------------------------------+----------+--------------+------------------------------+
root@asado:~#

keystone service-list show doesn't work:

root@asado:~# keystone service-get a51535b7482a4f9580fb5fdc44c24b93
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+-------------+----------------------------------+
| Property | Value |
+-------------+----------------------------------+
| description | OpenStack Volume Service |
| id | a51535b7482a4f9580fb5fdc44c24b93 |
| name | cinder |
| type | volume |
+-------------+----------------------------------+
root@asado:~# keystone service-get e8b017f0ce944df0b2a1785df1a3c3c6
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+-------------+----------------------------------+
| Property | Value |
+-------------+----------------------------------+
| description | OpenStack Volume Service |
| id | e8b017f0ce944df0b2a1785df1a3c3c6 |
| name | cinder ...

[Juan Pavlik (jjpavlik)]

Hi again, today i tried using 172.19.136.1 in my balancer node, where i balance API connections and i could make cinder work. That's because cinder is looking for services in 172.19.136.1, which is the first IP in the network (172.19.136.0/24), now i can list my volumes:

api-paste.ini:
[filter:authtoken]
paste.filter_factory = keystoneclient.middleware.auth_token:filter_factory
service_protocol = http
service_host = 172.19.136.10
service_port = 5000
auth_host = 172.19.136.10
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = cinder
admin_password = XXX
signing_dir = /var/lib/cinder

root@heladera:/etc/cinder# cinder --os-username XXX --os-tenant-name XXX --os-password XXX list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 43e7095d-5ef3-42c9-8053-3f46b46532ee | available | disco_normal | 2 | lvm | false | |
| 7c4679e9-d247-4dc7-98ca-04125409481d | available | disco_rapido | 2 | lvm_gold | false | |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
root@heladera:/etc/cinder#

So, everything is working fine, i just don't know why cinder is looking for the services in this IP.

[Sean McGinnis (sean-mcginnis)]

Closing stale bug. If this is still an issue please reopen.