glanceclient.exc.HTTPInternalServerError when running nova image-list

[Christoffer Brodd-Reijer]

I am following this guide verbatim: http://docs.openstack.org/liberty/install-guide-rdo/nova-verify.html (on CentOS 7.1).

When I run `nova image-list` I get the following output:

ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'glanceclient.exc.HTTPInternalServerError'> (HTTP 500) (Request-ID: req-d4a1d4b7-3bfd-46b5-bda7-37631106b839)

All other commands so far has worked fine. I can see services in nova and upload images in glance.

Logs attached.

[Christoffer Brodd-Reijer (ephracis-r)]

Attaching glance api log

[Christoffer Brodd-Reijer (ephracis-r)]

Attaching glance registry log

[Christoffer Brodd-Reijer (ephracis-r)]

FYI:

Firewall is disabled.

I have double-checked my configuration files and even rolled back and done the deployment twice, just to make sure there's not a typo or something that's messing things up.

[Christoffer Brodd-Reijer (ephracis-r)]

Here's my endpoints:

# openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------+
| 0ac56e9027834d1585be3e8f4f976daa | RegionOne | keystone | identity | True | admin | http://controller:5000/v2.0 |
| 2edff93b7945445b9a45bd70a341f2d2 | RegionOne | nova | compute | True | public | http://controller:8774/v2/%(tenant_id)s |
| 6dfbc51692294751aa3987478e70d6b7 | RegionOne | nova | compute | True | internal | http://controller:8774/v2/%(tenant_id)s |
| 7036db39b7994c36b8288b876573801a | RegionOne | neutron | network | True | admin | http://controller:9696 |
| 76b554158c73428ca77c487067d8cebd | RegionOne | keystone | identity | True | internal | http://controller:5000/v2.0 |
| 7953885128f34c94a496ba2b66836928 | RegionOne | neutron | network | True | internal | http://controller:9696 |
| 883df25fc1924786997e102e6d8371c3 | RegionOne | neutron | network | True | public | http://controller:9696 |
| 91b5f288e22e4d7da4a86d2452d266cb | RegionOne | nova | compute | True | admin | http://controller:8774/v2/%(tenant_id)s |
| bb1abe59e2d64389812ea165b68b909a | RegionOne | glance | image | True | internal | http://controller:9292 |
| dffc64f5f9e347f88eb9d1c1e6100a23 | RegionOne | glance | image | True | admin | http://controller:9292 |
| e0fb97c5714b4c1d9e3a8e0d5287620f | RegionOne | keystone | identity | True | public | http://controller:5000/v2.0 |
| ed95ce94bec0448f927d143ab33bdf5a | RegionOne | glance | image | True | public | http://controller:9292 |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------+

[Liyingjun (liyingjun)]

Couldn't reproduce on master.

[Fahri Cihan Demirci (cihand)]

From the logs I can see that glance failed to establish connection with keystone. Could it be possible that there was an temporary connection problem at that time?

[Hercules (sithiraklis)]

Ι concur! This thing has driven me nuts! Gone back and forth, checked configurations a thousand times and redone two fresh installations but still can't get it working :(
Attaching output of various commands, nova image-list and the error in nova-api.

Installed all prerequisites along with Identity, Glance, Compute and Networking components (and cinder but I truncated the logs for brevity since I think it's unrelated plus it works).

I'm probably going to ignore it for the time being but I hope it's not crucial for something to work...

P.S.: I'm aware that eventlet is deprecated and as such the openstack-keystone is deprecated but I see httpd is configured correctly so I think there is no point to use it since it's called from httpd using mod_wsgi.

[Hercules (sithiraklis)]

Forgot the nova-api.log :)

[Fahri Cihan Demirci (cihand)]

Regarding Keystone deprecation; only the Keystone CLI is deprecated and the other deprecation warnings you see have to do with the the major version transition if I'm not mistaken. However, I think there may be something wrong with your Keystone configuration, probably an incorrect auth-url setting, so that may be the reason why you are observing such a behaviour.

[Hercules (sithiraklis)]

Nope. Keystone configuration has been checked, rechecked and double checked. Besides, don't you find it strange that everything else works but specifically nova image-list fails with a HTTP 500? If it were something wrong with the configuration I believe keystone would totally fail in authentication.

P.S.: I'm attaching logs of glance-api and glance-registry. I get a service outage too. That's strange. What service outage? :P

[Fahri Cihan Demirci (cihand)]

Actually it seems to me that glance authentication already fails according to these logs. I think even the discovery of Keystone endpoints fails. What happens when you manually try to interact with Keystone, for example:

curl -s http://controller-win:5000/v2.0/

Normally you get something like the following:

{
  "version": {
    "status": "stable",
    "updated": "2014-04-17T00:00:00Z",
    "media-types": [
      {
        "base": "application/json",
        "type": "application/vnd.openstack.identity-v2.0+json"
      }
    ],
    "id": "v2.0",
    "links": [
      {
        "href": "http://127.0.0.1:5000/v2.0/",
        "rel": "self"
      },
      {
        "href": "http://docs.openstack.org/",
        "type": "text/html",
        "rel": "describedby"
      }
    ]
  }
}

[Hercules (sithiraklis)]

This is what I get:

{
  "version": {
    "status": "stable",
    "updated": "2014-04-17T00:00:00Z",
    "media-types": [
      {
        "base": "application/json",
        "type": "application/vnd.openstack.identity-v2.0+json"
      }
    ],
    "id": "v2.0",
    "links": [
      {
        "href": "http://controller-win:5000/v2.0/",
        "rel": "self"
      },
      {
        "href": "http://docs.openstack.org/",
        "type": "text/html",
        "rel": "describedby"
      }
    ]
  }
}

[Fahri Cihan Demirci (cihand)]

Well, that seems alright. However, Glance log definitely seems to point out to connection failures regarding Keystone. So I would guess that there is some intermittent network failure. I know this doesn't help you much but such a case is not easily reproducible, and rightly so because that would mean Keystone wasn't working correctly. So I would suggest changing your installation environment, maybe a different server or network. I'm hoping that you won't have the same problem.

[Hercules (sithiraklis)]

Ok, I'll try it. Maybe it has to do with the Hypervisor I'm using (Hyper-V - I know retarded but we want things to be cross-platform and we were evaluating its ... usefulness). I'll try VirtualBox and see if I have the same issues. I will report back with results.

[Sean Dague (sdague)]

I believe that this is the issue with Glance incompatibilities with v1 vs. v2 images. If you create an image with v2 via the glance API, glance explodes when you probe for that image over the v1 API because of property incompatibilities. Pushing back as a glance issue for now for that team to look into it.

[Kairat Kushaev (kkushaev)]

Just stupid questions (sorry for that):
What is keystone auth_url for v2:
I see from your comments it is "controller-win".
From glance registry it is:
2015-12-19 21:06:13.029 32438 WARNING keystoneclient.auth.identity.base [-] Failed to contact the endpoint at http://controller:5000/v2.0 for discovery. Fallback to using that endpoint as the base url - it is "controller".

[Hercules (sithiraklis)]

@kkushaev Where did you see that? I'm browsing through the glance registry logs and I can't find any line with the "controller" URL inside. Only with "controller-win". Are you looking at the right logs? :)

P.S.: There is nothing stupid in anything you ask. Maybe I've done something stupid but I've installed previous releases of OpenStack (Icehouse), I've re-done it at least 3 times the same setup over and over and I've searched meticulously all logs and settings again and again day after day with no result. I jut called it a day and now I'm automating the process since I had a script that it was making OpenStack images on CentOS 6.7 and I'm making a new for OpenStack Liberty using CentOS 7. Maybe I will be able to reproduce the bug this way consistently? :) :P

[Stuart McLaren (stuart-mclaren)]

@sdague

 >I believe that this is the issue with Glance incompatibilities with v1 vs. v2 images. If you create an image with v2 via the glance API, glance explodes when you probe for that image over the v1 API because of property incompatibilities.

Are you sure that's the right way around?

I know that we were having issues with images created in *v1* being listed by *v2* because the v2 listing used to try to validate the schema.

If there is an issue with v1 listing v2 created images do you have a link to more info?

Thanks.

[Stuart McLaren (stuart-mclaren)]

> All other commands so far has worked fine. I can see services in nova and upload images in glance

Specifically, which glance commands worked?

Does a 'glance image-list' work?
Does a 'glance image-show <id>' work?

It looks like the glance registry can't validate tokens:

2016-02-05 17:52:58.133 1845 CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to http://controller-win:5000/v2.0/auth/tokens
2016-02-05 17:52:58.190 1845 INFO eventlet.wsgi.server [-] 127.0.0.1 - - [05/Feb/2016 17:52:58] "GET /images/detail?is_public=none&limit=20 HTTP/1.1" 503 235 4.373681

If you send a curl command directly to the registry to list images what happens?

$ curl -v -H 'x-auth-token: <insert-a-real-token>' http://<registry-ip>/images/detail?is_public=none&limit=20'

what do the relevant glance-api.conf and glance-registry.conf configuration sections look like?

[Wenjun Wang (wangwenjun)]

@Christoffer Brodd-Reijer (ephracis-r):
I found this in your /var/log/glance/registry.log

"Unable to validate token: Unable to establish connection to http://controller:5000/v2.0/auth/tokens"

Have you solved this problem?

[Hercules (sithiraklis)]

@stuart-mclauren

>Does a 'glance image-list' work?
Does a 'glance image-show <id>' work?

Yes, they both work.

>If you send a curl command directly to the registry to list images what happens?
$ curl -v -H 'x-auth-token: <insert-a-real-token>' http://<registry-ip>/images/detail?is_public=none&limit=20'

I did:
openstack token issue
curl -v -H "x-auth-token: $ADMIN_TOKEN" http://controller-win:35357/images/detail?is_public=none&limit=20'

I get a HTTP 404 with message: '*Connection #0 to host controller-win left intact.' (and then the HTTP 404 error body)

P.S.: I sourced an admin-openrc.sh

[Stuart McLaren (stuart-mclaren)]

@suntao

Do you have a link to the fix? Thanks.