Horizon: Unable to launch instance

[Shannon McFarland]

As Edgar first reported and that several of us have confirmed, in our COE you cannot launch an instance from within the dashboard. When you submit the launch you get an error of "Error: There was an error submitting the form. Please try again."

Apache logs:
The apache logs are:
oot@p6-control01:~# [Thu Dec 06 18:06:54 2012] [error] unable to retrieve service catalog with token
[Thu Dec 06 18:06:54 2012] [error] Traceback (most recent call last):
[Thu Dec 06 18:06:54 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 132, in _extract_service_catalog
[Thu Dec 06 18:06:54 2012] [error] endpoint_type='adminURL')
[Thu Dec 06 18:06:54 2012] [error] File "/usr/lib/python2.7/dist-packages/keystoneclient/service_catalog.py", line 62, in url_for
[Thu Dec 06 18:06:54 2012] [error] raise exceptions.EndpointNotFound('Endpoint not found.')
[Thu Dec 06 18:06:54 2012] [error] EndpointNotFound: Endpoint not found.

Arvind wanted an output for:
root@control-01:/var/log/openvswitch# mysql -u root -p -e'use keystone;select * from endpoint\G';
Enter password:
*************************** 1. row ***************************
        id: 473eb4af320a44ef9bb7f9e73fae2b31
    region: RegionOne
service_id: c8bdf96141d344fdaa6ff03f1cc73f2a
     extra: {"adminurl": "http://10.121.13.50:8773/services/Admin", "internalurl": "http://10.121.13.50:8773/services/Cloud", "publicurl": "http://10.121.13.50:8773/services/Cloud"}
*************************** 2. row ***************************
        id: 56e62efe13fc4697b2e9d3b8a65ef922
    region: RegionOne
service_id: 1bece984d4184d24847c50a131498c2a
     extra: {"adminurl": "http://10.121.13.50:9696", "internalurl": "http://10.121.13.50:9696", "publicurl": "http://10.121.13.50:9696"}
*************************** 3. row ***************************
        id: bb0f0aa0b3374e98b308c636b53be24d
    region: RegionOne
service_id: 6f6226cc2c7f485881a6623f59454a97
     extra: {"adminurl": "http://10.121.13.50:8776/v1/%(tenant_id)s", "internalurl": "http://10.121.13.50:8776/v1/%(tenant_id)s", "publicurl": "http://10.121.13.50:8776/v1/%(tenant_id)s"}
*************************** 4. row ***************************
        id: c2993e557dbf49628c15c7537db78141
    region: RegionOne
service_id: 80ec9fdb738e453f94a9e7ac7af44dcc
     extra: {"adminurl": "http://10.121.13.50:8774/v2/%(tenant_id)s", "internalurl": "http://10.121.13.50:8774/v2/%(tenant_id)s", "publicurl": "http://10.121.13.50:8774/v2/%(tenant_id)s"}
*************************** 5. row ***************************
        id: cbac767e7510465fb80f972c47a7680d
    region: RegionOne
service_id: e5a575f1ab774c3e8acda354cc5bcab8
     extra: {"adminurl": "http://10.121.13.50:9292/v1", "internalurl": "http://10.121.13.50:9292/v1", "publicurl": "http://10.121.13.50:9292/v1"}
*************************** 6. row ***************************
        id: d0bb1f83ccf04d9289bef0febd532a04
    region: RegionOne
service_id: e79f91e387f04cdea8d19b3178c0fd39
     extra: {"adminurl": "http://10.121.13.50:35357/v2.0", "internalurl": "http://10.121.13.50:5000/v2.0", "publicurl": "http://10.121.13.50:5000/v2.0"}

[Arvind Somya (asomya)]

I have traced this issue to Horizon not passing the tenant id or name to keystone, this leads to keystone reporting blank service catalogs and the subsequent endpoint failure. Investigating whether this is screwed up in our repos or globally in folsom.

[Arvind Somya (asomya)]

Scratch the previous comment, Just managed to get my own setup working in debug mode and it looks like a Quantum/QuantumManager issue.

[Thu Dec 13 15:01:14 2012] [error]
[Thu Dec 13 15:01:14 2012] [error] ERROR:django.request:Internal Server Error: /nova/instances/launch
[Thu Dec 13 15:01:14 2012] [error] Traceback (most recent call last):
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
[Thu Dec 13 15:01:14 2012] [error] response = callback(request, *callback_args, **callback_kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 40, in dec
[Thu Dec 13 15:01:14 2012] [error] return view_func(request, *args, **kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 55, in dec
[Thu Dec 13 15:01:14 2012] [error] return view_func(request, *args, **kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/decorators.py", line 40, in dec
[Thu Dec 13 15:01:14 2012] [error] return view_func(request, *args, **kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 48, in view
[Thu Dec 13 15:01:14 2012] [error] return self.dispatch(request, *args, **kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/django/views/generic/base.py", line 69, in dispatch
[Thu Dec 13 15:01:14 2012] [error] return handler(request, *args, **kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/workflows/views.py", line 139, in post
[Thu Dec 13 15:01:14 2012] [error] exceptions.handle(request)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/workflows/views.py", line 136, in post
[Thu Dec 13 15:01:14 2012] [error] success = workflow.finalize()
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/workflows/base.py", line 748, in finalize
[Thu Dec 13 15:01:14 2012] [error] if not self.handle(self.request, self.context):
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/dashboards/nova/instances/workflows.py", line 509, in handle
[Thu Dec 13 15:01:14 2012] [error] exceptions.handle(request)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/dashboards/nova/instances/workflows.py", line 506, in handle
[Thu Dec 13 15:01:14 2012] [error] instance_count=int(context['count']))
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/horizon/api/nova.py", line 305, in server_create
[Thu Dec 13 15:01:14 2012] [error] min_count=instance_count), request)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/servers.py", line 498, in create
[Thu Dec 13 15:01:14 2012] [error] **boot_kwargs)
[Thu Dec 13 15:01:14 2012] [error] File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/base.py", line 153, in _boot
[Thu Dec 13 15:01:14 2012] [error] if nic_info['port-id']:
[Thu Dec 13 15:0...

[Arvind Somya (asomya)]

Issue pinpointed to a bug in python-novaclient, this bug has since been fixed upstream.

We need to update our repository and rebuild the package. I have synced our python-novaclient repository to upstream.

[Ian Wells (ijw-ubuntu)]

Note that running from the CLI does work - it's only Horizon that has an issue.

[Mark T. Voelker (mvoelker)]

@Arvind: So is this now fixed? Can we mark it as such? Or are we waiting on packaging?

[Robert Starmer (starmer)]

Still not fixed. Targeting our current repo (ftpeng and/or http target) and getting the same issue.

[Robert Starmer (starmer)]

Short term fix is part of the following pull request:

https://github.com/CiscoSystems/folsom-manifests/pull/27

[Arvind Somya (asomya)]

Fixed, Replaced the old folsom branch with the folsom-1 tag in the python-novaclient repository that contained a fix for this issue. Our CI has built new packages and Robert Starmer confirms the new modules work without the patch.