Openstack services return 401 (glance, cinder, nova), deployment fails to upload cirros image

[Andrew Woodward]

{"build_id": "2014-04-18_18-05-22", "mirantis": "yes", "build_number": "113", "nailgun_sha": "0b10ed5fae7a8b140917c6b5381e1d14a2b0e727", "production": "prod", "ostf_sha": "301cbbc38f71739328f04f3c421f1272492589dd", "fuelmain_sha": "f0b6f3afe87b0b286d22ee6114b92c1c07fd1ce6", "astute_sha": "0cb0b26773bd269000a14748e8ba49419462ecaf", "release": "5.0", "fuellib_sha": "61801bf39dfe19b85877cbea00efeedd36dd8d57"}

deployment of ubuntu, ha, neutron vlan
Ceph for images, volumes, compute, object

3 controller, 2 compute + ceph

deployment fails to upload cirros image

[5033] 37023a82-70ef-4c1c-a0a5-71c9efa776d4: cmd: /usr/bin/glance -N http://192.168.0.2:5000/v2.0/ -T admin -I admin -K admin image-create --name 'TestVM' --is-public true --container-format='bare' --disk-format='qcow2' --property murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}' --file '/usr/share/cirros-testvm/cirros-x86_64-disk.img'
                                               stdout:
                                               stderr: Request returned failure status.
Invalid OpenStack Identity credentials.
                                               exit code: 1

root@node-1:~# source openrc
root@node-1:~# cinder create 1
ERROR: Unauthorized (HTTP 401)
root@node-1:~# glance index
ID Name Disk Format Container Format Size
------------------------------------ ------------------------------ -------------------- -------------------- --------------
Request returned failure status.
Invalid OpenStack Identity credentials.
root@node-1:~# nova service-list
ERROR: Unauthorized (HTTP 401)

root@node-1:~# nova --debug service-list

REQ: curl -i 'http://192.168.0.2:5000/v2.0/tokens' -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

New session created for: (http://192.168.0.2:5000)
INFO (connectionpool:202) Starting new HTTP connection (1): 192.168.0.2
DEBUG (connectionpool:296) "POST /v2.0/tokens HTTP/1.1" 200 3092
RESP: [200] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:18:58 GMT', 'vary': 'X-Auth-Token', 'content-length': '3092', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-04-18T22:18:58.544566", "expires": "2014-04-18T23:18:58Z", "id": "4bb9face939d40df82446a88cd51a6c1", "tenant": {"description": "admin tenant", "enabled": true, "id": "7e9f34f9aa9a481388082193392439e1", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "id": "d68f2e04d90f4fdaa5eabcdffd78d37e", "publicURL": "http://10.108.38.2:8774/v2/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9696", "region": "RegionOne", "internalURL": "http://192.168.0.2:9696", "id": "fd3cd1528757469ea9c118723fb29404", "publicURL": "http://10.108.38.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9292", "region": "RegionOne", "internalURL": "http://192.168.0.2:9292", "id": "f928c228a143434fa2a0ffd969208a8e", "publicURL": "http://10.108.38.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "id": "27e3591c40d1453a9fae5ea5c79b474e", "publicURL": "http://10.108.38.2:8776/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.2:8773/services/Cloud", "id": "d209d1ab705143db9781e08c56abff58", "publicURL": "http://10.108.38.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "id": "e10f707b3d474b98b83ec66282adc8e7", "publicURL": "http://10.108.38.2:8004/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8080/swift/v1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8080/swift/v1", "id": "f31a44c4c09a4cbbb1d4fe41d9b53551", "publicURL": "http://10.108.38.2:8080/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.2:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.2:5000/v2.0", "id": "70c11159a1ad48309828ed2775b9d3d4", "publicURL": "http://10.108.38.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "e33163b892324ca5983998514ab14949", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "a85159d66c6146b5ae735093f0b3bbc1"]}}}

REQ: curl -i 'http://10.108.38.2:8774/v2/7e9f34f9aa9a481388082193392439e1/os-services' -X GET -H "X-Auth-Project-Id: admin" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: 4bb9face939d40df82446a88cd51a6c1"

New session created for: (http://10.108.38.2:8774)
INFO (connectionpool:202) Starting new HTTP connection (1): 10.108.38.2
DEBUG (connectionpool:296) "GET /v2/7e9f34f9aa9a481388082193392439e1/os-services HTTP/1.1" 401 23
RESP: [401] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:18:58 GMT', 'content-length': '23', 'content-type': 'text/plain', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357'"})
RESP BODY: Authentication required

REQ: curl -i 'http://192.168.0.2:5000/v2.0/tokens' -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-novaclient" -d '{"auth": {"passwordCredentials": {"username": "admin", "password": "admin"}, "tenantId": "7e9f34f9aa9a481388082193392439e1"}}'

New session created for: (http://192.168.0.2:5000)
INFO (connectionpool:202) Starting new HTTP connection (1): 192.168.0.2
DEBUG (connectionpool:296) "POST /v2.0/tokens HTTP/1.1" 401 161
RESP: [401] CaseInsensitiveDict({'content-length': '161', 'vary': 'X-Auth-Token', 'www-authenticate': 'Keystone uri="http://192.168.0.2:5000"', 'date': 'Fri, 18 Apr 2014 21:46:47 GMT', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"error": {"message": "User 3fc0b86a98e74173b7225c0741f14504 is unauthorized for tenant 7e9f34f9aa9a481388082193392439e1", "code": 401, "title": "Unauthorized"}}

DEBUG (shell:777) Unauthorized (HTTP 401)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 774, in main
    OpenStackComputeShell().main(map(strutils.safe_decode, sys.argv[1:]))
  File "/usr/lib/python2.7/dist-packages/novaclient/shell.py", line 710, in main
    args.func(self.cs, args)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/shell.py", line 2853, in do_service_list
    result = cs.services.list(host=args.host, binary=args.binary)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/services.py", line 49, in list
    return self._list(url, "services")
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 275, in _cs_request
    raise e
Unauthorized: Unauthorized (HTTP 401)
ERROR: Unauthorized (HTTP 401)
root@node-1:~#
root@node-1:~#
root@node-1:~# cinder --debug create 1

REQ: curl -i http://192.168.0.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

DEBUG:cinderclient.client:
REQ: curl -i http://192.168.0.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

RESP: [200] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:10 GMT', 'vary': 'X-Auth-Token', 'content-length': '3092', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-04-18T22:19:10.734604", "expires": "2014-04-18T23:19:10Z", "id": "ba8d814d21844126b204048ebec61ebc", "tenant": {"description": "admin tenant", "enabled": true, "id": "693f8b53ab3e40648d0b4b6028d2db19", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19", "id": "8ddb0b264a6b495398d3c14e3e8ab0c3", "publicURL": "http://10.108.38.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9696", "region": "RegionOne", "internalURL": "http://192.168.0.2:9696", "id": "135c7493e67042b0a0af9397a34d0c94", "publicURL": "http://10.108.38.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9292", "region": "RegionOne", "internalURL": "http://192.168.0.2:9292", "id": "120d98e1c2394de0b78a91329ac7208d", "publicURL": "http://10.108.38.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19", "id": "5c1879a4b9224c5893ec85fa49bcd45a", "publicURL": "http://10.108.38.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.2:8773/services/Cloud", "id": "ac0a37148b274cb0bb5d2ff798839008", "publicURL": "http://10.108.38.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19", "id": "704db6ab06684adab2ba0382a26b3547", "publicURL": "http://10.108.38.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8080/swift/v1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8080/swift/v1", "id": "064027f83da04665aef9ed1ea9ddb654", "publicURL": "http://10.108.38.2:8080/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.2:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.2:5000/v2.0", "id": "0d9a71a153a640569aee5bc67ff50120", "publicURL": "http://10.108.38.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "b7dfa2ab4fe24389805dc4e653463773", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "0c9ea658b5b64c65adebf08c8a55893a"]}}}

DEBUG:cinderclient.client:RESP: [200] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:10 GMT', 'vary': 'X-Auth-Token', 'content-length': '3092', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-04-18T22:19:10.734604", "expires": "2014-04-18T23:19:10Z", "id": "ba8d814d21844126b204048ebec61ebc", "tenant": {"description": "admin tenant", "enabled": true, "id": "693f8b53ab3e40648d0b4b6028d2db19", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19", "id": "8ddb0b264a6b495398d3c14e3e8ab0c3", "publicURL": "http://10.108.38.2:8774/v2/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9696", "region": "RegionOne", "internalURL": "http://192.168.0.2:9696", "id": "135c7493e67042b0a0af9397a34d0c94", "publicURL": "http://10.108.38.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9292", "region": "RegionOne", "internalURL": "http://192.168.0.2:9292", "id": "120d98e1c2394de0b78a91329ac7208d", "publicURL": "http://10.108.38.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19", "id": "5c1879a4b9224c5893ec85fa49bcd45a", "publicURL": "http://10.108.38.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.2:8773/services/Cloud", "id": "ac0a37148b274cb0bb5d2ff798839008", "publicURL": "http://10.108.38.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19", "region": "RegionOne", "internalURL": "http://192.168.0.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19", "id": "704db6ab06684adab2ba0382a26b3547", "publicURL": "http://10.108.38.2:8004/v1/693f8b53ab3e40648d0b4b6028d2db19"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8080/swift/v1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8080/swift/v1", "id": "064027f83da04665aef9ed1ea9ddb654", "publicURL": "http://10.108.38.2:8080/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.2:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.2:5000/v2.0", "id": "0d9a71a153a640569aee5bc67ff50120", "publicURL": "http://10.108.38.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "b7dfa2ab4fe24389805dc4e653463773", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "0c9ea658b5b64c65adebf08c8a55893a"]}}}

REQ: curl -i http://10.108.38.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19/volumes -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: ba8d814d21844126b204048ebec61ebc" -d '{"volume": {"status": "creating", "availability_zone": null, "source_volid": null, "display_description": null, "snapshot_id": null, "user_id": null, "size": 1, "display_name": null, "imageRef": null, "attach_status": "detached", "volume_type": null, "project_id": null, "metadata": {}}}'

DEBUG:cinderclient.client:
REQ: curl -i http://10.108.38.2:8776/v1/693f8b53ab3e40648d0b4b6028d2db19/volumes -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: ba8d814d21844126b204048ebec61ebc" -d '{"volume": {"status": "creating", "availability_zone": null, "source_volid": null, "display_description": null, "snapshot_id": null, "user_id": null, "size": 1, "display_name": null, "imageRef": null, "attach_status": "detached", "volume_type": null, "project_id": null, "metadata": {}}}'

RESP: [401] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'content-length': '23', 'content-type': 'text/plain', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357'", 'x-openstack-request-id': 'req-c16f0243-4c76-496f-8aa6-c181ec772a1e'})
RESP BODY: Authentication required

DEBUG:cinderclient.client:RESP: [401] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'content-length': '23', 'content-type': 'text/plain', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357'", 'x-openstack-request-id': 'req-c16f0243-4c76-496f-8aa6-c181ec772a1e'})
RESP BODY: Authentication required

Unauthorized, reauthenticating.
DEBUG:cinderclient.client:Unauthorized, reauthenticating.

REQ: curl -i http://192.168.0.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

DEBUG:cinderclient.client:
REQ: curl -i http://192.168.0.2:5000/v2.0/tokens -X POST -H "Content-Type: application/json" -H "Accept: application/json" -H "User-Agent: python-cinderclient" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}'

RESP: [200] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'vary': 'X-Auth-Token', 'content-length': '3092', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-04-18T22:19:11.287450", "expires": "2014-04-18T23:19:11Z", "id": "8e9ff123dc5b4904bf4acfcab01eb7da", "tenant": {"description": "admin tenant", "enabled": true, "id": "7e9f34f9aa9a481388082193392439e1", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "id": "d68f2e04d90f4fdaa5eabcdffd78d37e", "publicURL": "http://10.108.38.2:8774/v2/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9696", "region": "RegionOne", "internalURL": "http://192.168.0.2:9696", "id": "fd3cd1528757469ea9c118723fb29404", "publicURL": "http://10.108.38.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9292", "region": "RegionOne", "internalURL": "http://192.168.0.2:9292", "id": "f928c228a143434fa2a0ffd969208a8e", "publicURL": "http://10.108.38.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "id": "27e3591c40d1453a9fae5ea5c79b474e", "publicURL": "http://10.108.38.2:8776/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.2:8773/services/Cloud", "id": "d209d1ab705143db9781e08c56abff58", "publicURL": "http://10.108.38.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "id": "e10f707b3d474b98b83ec66282adc8e7", "publicURL": "http://10.108.38.2:8004/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8080/swift/v1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8080/swift/v1", "id": "f31a44c4c09a4cbbb1d4fe41d9b53551", "publicURL": "http://10.108.38.2:8080/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.2:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.2:5000/v2.0", "id": "70c11159a1ad48309828ed2775b9d3d4", "publicURL": "http://10.108.38.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "e33163b892324ca5983998514ab14949", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "a85159d66c6146b5ae735093f0b3bbc1"]}}}

DEBUG:cinderclient.client:RESP: [200] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'vary': 'X-Auth-Token', 'content-length': '3092', 'content-type': 'application/json', 'x-distribution': 'Ubuntu'})
RESP BODY: {"access": {"token": {"issued_at": "2014-04-18T22:19:11.287450", "expires": "2014-04-18T23:19:11Z", "id": "8e9ff123dc5b4904bf4acfcab01eb7da", "tenant": {"description": "admin tenant", "enabled": true, "id": "7e9f34f9aa9a481388082193392439e1", "name": "admin"}}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8774/v2/7e9f34f9aa9a481388082193392439e1", "id": "d68f2e04d90f4fdaa5eabcdffd78d37e", "publicURL": "http://10.108.38.2:8774/v2/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "compute", "name": "nova"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9696", "region": "RegionOne", "internalURL": "http://192.168.0.2:9696", "id": "fd3cd1528757469ea9c118723fb29404", "publicURL": "http://10.108.38.2:9696"}], "endpoints_links": [], "type": "network", "name": "neutron"}, {"endpoints": [{"adminURL": "http://192.168.0.2:9292", "region": "RegionOne", "internalURL": "http://192.168.0.2:9292", "id": "f928c228a143434fa2a0ffd969208a8e", "publicURL": "http://10.108.38.2:9292"}], "endpoints_links": [], "type": "image", "name": "glance"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8776/v1/7e9f34f9aa9a481388082193392439e1", "id": "27e3591c40d1453a9fae5ea5c79b474e", "publicURL": "http://10.108.38.2:8776/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "volume", "name": "cinder"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8773/services/Admin", "region": "RegionOne", "internalURL": "http://192.168.0.2:8773/services/Cloud", "id": "d209d1ab705143db9781e08c56abff58", "publicURL": "http://10.108.38.2:8773/services/Cloud"}], "endpoints_links": [], "type": "ec2", "name": "nova_ec2"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8004/v1/7e9f34f9aa9a481388082193392439e1", "id": "e10f707b3d474b98b83ec66282adc8e7", "publicURL": "http://10.108.38.2:8004/v1/7e9f34f9aa9a481388082193392439e1"}], "endpoints_links": [], "type": "orchestration", "name": "heat"}, {"endpoints": [{"adminURL": "http://192.168.0.2:8080/swift/v1", "region": "RegionOne", "internalURL": "http://192.168.0.2:8080/swift/v1", "id": "f31a44c4c09a4cbbb1d4fe41d9b53551", "publicURL": "http://10.108.38.2:8080/swift/v1"}], "endpoints_links": [], "type": "object-store", "name": "swift"}, {"endpoints": [{"adminURL": "http://192.168.0.2:35357/v2.0", "region": "RegionOne", "internalURL": "http://192.168.0.2:5000/v2.0", "id": "70c11159a1ad48309828ed2775b9d3d4", "publicURL": "http://10.108.38.2:5000/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "e33163b892324ca5983998514ab14949", "roles": [{"name": "_member_"}, {"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["9fe2ff9ee4384b1894a90878d3e92bab", "a85159d66c6146b5ae735093f0b3bbc1"]}}}

REQ: curl -i http://10.108.38.2:8776/v1/7e9f34f9aa9a481388082193392439e1/volumes -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: 8e9ff123dc5b4904bf4acfcab01eb7da" -d '{"volume": {"status": "creating", "availability_zone": null, "source_volid": null, "display_description": null, "snapshot_id": null, "user_id": null, "size": 1, "display_name": null, "imageRef": null, "attach_status": "detached", "volume_type": null, "project_id": null, "metadata": {}}}'

DEBUG:cinderclient.client:
REQ: curl -i http://10.108.38.2:8776/v1/7e9f34f9aa9a481388082193392439e1/volumes -X POST -H "X-Auth-Project-Id: admin" -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: 8e9ff123dc5b4904bf4acfcab01eb7da" -d '{"volume": {"status": "creating", "availability_zone": null, "source_volid": null, "display_description": null, "snapshot_id": null, "user_id": null, "size": 1, "display_name": null, "imageRef": null, "attach_status": "detached", "volume_type": null, "project_id": null, "metadata": {}}}'

RESP: [401] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'content-length': '23', 'content-type': 'text/plain', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357'", 'x-openstack-request-id': 'req-418e836c-1559-4983-8ce3-ee98c1ed375a'})
RESP BODY: Authentication required

DEBUG:cinderclient.client:RESP: [401] CaseInsensitiveDict({'date': 'Fri, 18 Apr 2014 22:19:11 GMT', 'content-length': '23', 'content-type': 'text/plain', 'www-authenticate': "Keystone uri='http://192.168.0.2:35357'", 'x-openstack-request-id': 'req-418e836c-1559-4983-8ce3-ee98c1ed375a'})
RESP BODY: Authentication required

ERROR: Unauthorized (HTTP 401)

root@node-1:~# keystone user-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+---------+---------+-------------------+
| id | name | enabled | email |
+----------------------------------+---------+---------+-------------------+
| b7dfa2ab4fe24389805dc4e653463773 | admin | True | <email address hidden> |
| db253bbe2ef240e796f3c47d2b02f926 | cinder | True | cinder@localhost |
| c407c3ed06f34a3491c293da7601c803 | glance | True | glance@localhost |
| e881a072ba1945c28be9bf2c63c5a779 | heat | True | <email address hidden> |
| 32cd9492070042139bf8abf8f1da10c9 | neutron | True | neutron@localhost |
| bf0226bedb3e434c91b81b56daac117d | nova | True | nova@localhost |
+----------------------------------+---------+---------+-------------------+
root@node-1:~# keystone tenant-list
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).
+----------------------------------+----------+---------+
| id | name | enabled |
+----------------------------------+----------+---------+
| 8480f0f980a242a8941789a31a2503c0 | admin | True |
| 6d7029b9574b41479ff1f4d2afbf6161 | services | True |
+----------------------------------+----------+---------+
root@node-1:~#

root@node-1:/etc/keystone# grep ^connection keystone*
keystone.conf:connection = sqlite:////var/lib/keystone/keystone.db
keystone.conf:connection=mysql://keystone:oGYpKK2T@192.168.0.2/keystone?read_timeout=60
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2
ERROR 1045 (28000): Access denied for user 'keystone'@'192.168.0.2' (using password: NO)
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2 -p oGYpKK2T
Enter password:
ERROR 1044 (42000): Access denied for user 'keystone'@'%' to database 'oGYpKK2T'
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2
ERROR 1045 (28000): Access denied for user 'keystone'@'192.168.0.2' (using password: NO)
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2286
Server version: 5.5.28-23.7ubuntu0.12.04.2+mirantis.wsrep3 (Ubuntu), wsrep_23.7.rXXXX

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use keystone;
Database changed
mysql> show tables;
Empty set (0.00 sec)

mysql> Bye
root@node-1:/etc/keystone# cd /var/lib/keystone/
root@node-1:/var/lib/keystone# ls
cache keystone.db
root@node-1:/var/lib/keystone# ll
total 592
drwxr-xr-x 3 keystone keystone 4096 Apr 18 21:56 ./
drwxr-xr-x 58 root root 4096 Apr 18 18:27 ../
drwxr-xr-x 2 root root 4096 Apr 10 14:31 cache/
-rw-r--r-- 1 keystone keystone 584704 Apr 18 21:56 keystone.db
-rw------- 1 keystone keystone 1024 Apr 18 18:24 .rnd
root@node-1:/var/lib/keystone#

root@node-1:/etc/keystone# grep ^connection keystone*
keystone.conf:connection = sqlite:////var/lib/keystone/keystone.db
keystone.conf:connection=mysql://keystone:oGYpKK2T@192.168.0.2/keystone?read_timeout=60
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2
ERROR 1045 (28000): Access denied for user 'keystone'@'192.168.0.2' (using password: NO)
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2 -p oGYpKK2T
Enter password:
ERROR 1044 (42000): Access denied for user 'keystone'@'%' to database 'oGYpKK2T'
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2
ERROR 1045 (28000): Access denied for user 'keystone'@'192.168.0.2' (using password: NO)
root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2286
Server version: 5.5.28-23.7ubuntu0.12.04.2+mirantis.wsrep3 (Ubuntu), wsrep_23.7.rXXXX

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use keystone;
Database changed
mysql> show tables;
Empty set (0.00 sec)

mysql> Bye
root@node-1:/etc/keystone# cd /var/lib/keystone/
root@node-1:/var/lib/keystone# ls
cache keystone.db
root@node-1:/var/lib/keystone# ll
total 592
drwxr-xr-x 3 keystone keystone 4096 Apr 18 21:56 ./
drwxr-xr-x 58 root root 4096 Apr 18 18:27 ../
drwxr-xr-x 2 root root 4096 Apr 10 14:31 cache/
-rw-r--r-- 1 keystone keystone 584704 Apr 18 21:56 keystone.db
-rw------- 1 keystone keystone 1024 Apr 18 18:24 .rnd
root@node-1:/var/lib/keystone#

root@node-1:/etc/keystone# grep ^connection keystone*
keystone.conf:connection = sqlite:////var/lib/keystone/keystone.db
keystone.conf:connection=mysql://keystone:oGYpKK2T@192.168.0.2/keystone?read_timeout=60

root@node-1:/etc/keystone# mysql -u keystone -h 192.168.0.2 -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2286
Server version: 5.5.28-23.7ubuntu0.12.04.2+mirantis.wsrep3 (Ubuntu), wsrep_23.7.rXXXX

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use keystone;
Database changed
mysql> show tables;
Empty set (0.00 sec)

mysql> Bye
root@node-1:/etc/keystone# cd /var/lib/keystone/
root@node-1:/var/lib/keystone# ls
cache keystone.db
root@node-1:/var/lib/keystone# ll
total 592
drwxr-xr-x 3 keystone keystone 4096 Apr 18 21:56 ./
drwxr-xr-x 58 root root 4096 Apr 18 18:27 ../
drwxr-xr-x 2 root root 4096 Apr 10 14:31 cache/
-rw-r--r-- 1 keystone keystone 584704 Apr 18 21:56 keystone.db
-rw------- 1 keystone keystone 1024 Apr 18 18:24 .rnd
root@node-1:/var/lib/keystone#

the issue appears to be the ini_file provider not putting a space on each side of the = assignment and creating duplicate entries.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/88739

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88739
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7f0606a5d6601c814492f0fc8e5cb399792ef1c3
Submitter: Jenkins
Branch: master

commit 7f0606a5d6601c814492f0fc8e5cb399792ef1c3
Author: Andrew Woodward <email address hidden>
Date: Fri Apr 18 17:18:04 2014 -0700

    Fix db settings for keystone with relation to icehouse

    The sql section is no longer valid, other options no longer have
     the sql_ predicate.

    https://github.com/openstack/keystone/blob/master/keystone/openstack/common/db/options.py

    Change-Id: I1a31686bd878f96e1a6a73b8f9812719afd1af9f
    Resolves-bug: #1309780

[Anastasia Palkina (apalkina)]

Verified on ISO #117
"build_id": "2014-04-21_12-12-37",
"mirantis": "yes",
"build_number": "117",
"nailgun_sha": "66395c99412dbd893868d428b0e4748be70e4a61",
"production": "prod",
"ostf_sha": "7d0ea20e1c450a5f571d91bb03141b12c301184b",
"fuelmain_sha": "f0b6f3afe87b0b286d22ee6114b92c1c07fd1ce6",
"astute_sha": "0cb0b26773bd269000a14748e8ba49419462ecaf",
"release": "5.0",
"fuellib_sha": "b0390f0f882b4f1a3969c16f0514761c81691cc2"

[Vasilios Tzanoudakis (vtzanoudakis)]

Same bug here on fuel-community-5.1-31-2014-09-18_06-04-08.iso

http://paste.openstack.org/show/112810/

[Vasilios Tzanoudakis (vtzanoudakis)]

Running the failed command by hand worked although. # /usr/bin/glance image-create --name 'TestVM1' --is-public true --container-format='bare' --disk-format='qcow2' --min-ram=64 --property murano_image_info='{"title": "Murano Demo", "type": "cirros.demo"}'