openstack：Pike
# tailf /var/log/nova/nova-api.log
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions self.flavor = instance.flavor
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 67, in getter
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions self.obj_load_attr(name)
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/objects/instance.py", line 1029, in obj_load_attr
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions objtype=self.obj_name())
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions OrphanedObjectError: Cannot call obj_load_attr on orphaned Instance object
2018-05-30 14:00:34.832 170348 ERROR nova.api.openstack.extensions
2018-05-30 14:00:34.834 170348 INFO nova.api.openstack.wsgi [req-dde33c8b-7d91-4938-bab2-0410c046dd71 516e4eeaf0614802b3af422f40b140b6 befcf911008745b69ca9e4c1fd1e868e - default default] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.OrphanedObjectError'>
2018-05-30 14:00:34.836 170348 INFO nova.osapi_compute.wsgi.server [req-dde33c8b-7d91-4938-bab2-0410c046dd71 516e4eeaf0614802b3af422f40b140b6 befcf911008745b69ca9e4c1fd1e868e - default default] 172.20.239.50 "GET /v2.1/servers/detail HTTP/1.1" status: 500 len: 577 time: 0.2061019
# nova --debug list
DEBUG (extension:180) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:180) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:180) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:180) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:180) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:180) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (session:347) REQ: curl -g -i -X GET http://controller:35357/v3 -H "Accept: application/json" -H "User-Agent: nova keystoneauth1/2.18.0 python-requests/2.11.1 CPython/2.7.5"
INFO (connectionpool:214) Starting new HTTP connection (1): controller
DEBUG (connectionpool:401) "GET /v3 HTTP/1.1" 200 250
DEBUG (session:395) RESP: [200] Date: Wed, 30 May 2018 06:03:55 GMT Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token x-openstack-request-id: req-21122cf8-a4c6-459e-9432-c6715297effd Content-Length: 250 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "http://controller:35357/v3/", "rel": "self"}]}}
DEBUG (session:640) GET call to None for http://controller:35357/v3 used request id req-21122cf8-a4c6-459e-9432-c6715297effd
DEBUG (base:165) Making authentication request to http://controller:35357/v3/auth/tokens
DEBUG (connectionpool:401) "POST /v3/auth/tokens HTTP/1.1" 201 4596
DEBUG (base:170) {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "c830219ed3884d61bb0109a04450adc5", "name": "admin"}], "expires_at": "2018-05-30T07:03:55.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "befcf911008745b69ca9e4c1fd1e868e", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://controller:9292", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "39438cc019114f03b063c73ca251c363"}, {"url": "http://controller:9292", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "93e35640c1a7443a83109d2f7d34a411"}, {"url": "http://controller:9292", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ae1e31f0eb344d01a2ca7dfcb83c799c"}], "type": "image", "id": "00ed14d3fdbe44b3ad2141a0707ccc60", "name": "glance"}, {"endpoints": [{"url": "http://controller:8776/v1/befcf911008745b69ca9e4c1fd1e868e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "2a6ef00113654a1bb989061006919b96"}, {"url": "http://controller:8776/v1/befcf911008745b69ca9e4c1fd1e868e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "4a5c2dd92b1045a9b901f2f4dfab701e"}, {"url": "http://controller:8776/v1/befcf911008745b69ca9e4c1fd1e868e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "888f9a88dfa14b5187aec180d6ad193a"}], "type": "volume", "id": "6d7b1212e7484b09b7b760ead0653adc", "name": "cinder"}, {"endpoints": [{"url": "http://controller:8778", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "338bce72a6424939af67e6141c50e6cf"}, {"url": "http://controller:8778", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "a42848cc730842619f0024adbe4b21c7"}, {"url": "http://controller:8778", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "b184f681e7d84ef3b64180931e34eed2"}], "type": "placement", "id": "7cead6f4be0443ffb540da9915a5b137", "name": "placement"}, {"endpoints": [{"url": "http://controller:9696", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "1cf1f55e43074cc8bc63ef9a48d98953"}, {"url": "http://controller:9696", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "46aec5a91a0441a48c11a70add67a119"}, {"url": "http://controller:9696", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "ecbca7ce51cb41a086707bb5b79dfc88"}], "type": "network", "id": "86ebb117e2d04fa6bb9bb07787168530", "name": "neutron"}, {"endpoints": [{"url": "http://controller:8776/v2/befcf911008745b69ca9e4c1fd1e868e", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "2835d797edd344f2980479a89dec2dc3"}, {"url": "http://controller:8776/v2/befcf911008745b69ca9e4c1fd1e868e", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "4aa4363fd3d1418db40d2a2ed371bda5"}, {"url": "http://controller:8776/v2/befcf911008745b69ca9e4c1fd1e868e", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "f4d4238aa46040d1a3fcf7ab83b0da5b"}], "type": "volumev2", "id": "ad5e19a43c564beab2b09d86f665dbf0", "name": "cinderv2"}, {"endpoints": [{"url": "http://controller:5000/v3/", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "52f20dce6e5a4517abf479e1adb23ad5"}, {"url": "http://controller:35357/v3/", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "73e9b49a5c264eaf8a62a4414dbc16f4"}, {"url": "http://controller:5000/v3/", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "d807ec87a2c9451588c97f93a5e16a92"}], "type": "identity", "id": "da766ecfa565438fafd0fad8469529e0", "name": "keystone"}, {"endpoints": [{"url": "http://controller:8774/v2.1", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "483ff7a94bf143649c177766a1712d0e"}, {"url": "http://controller:8774/v2.1", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f549cec9b9af4c1fbc675349ea93a43f"}, {"url": "http://controller:8774/v2.1", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f9dbac0154cd4a21ba0c8876d33d050c"}], "type": "compute", "id": "f9b1e8d3f6a14fd8a4154966e27f88d7", "name": "nova"}], "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "516e4eeaf0614802b3af422f40b140b6"}, "audit_ids": ["lBvcWuGZTfG61O2RpBwzJA"], "issued_at": "2018-05-30T06:03:55.000000Z"}}
REQ: curl -g -i -X GET http://controller:8774/v2.1 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f7a1d3aa256bf97008b37c21c1b4b702bac62158"
DEBUG (session:347) REQ: curl -g -i -X GET http://controller:8774/v2.1 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}f7a1d3aa256bf97008b37c21c1b4b702bac62158"
INFO (connectionpool:214) Starting new HTTP connection (1): controller
DEBUG (connectionpool:401) "GET /v2.1 HTTP/1.1" 302 0
RESP: [302] Content-Type: text/plain; charset=utf8 Location: http://controller:8774/v2.1/ X-Compute-Request-Id: req-4ff0c878-32a4-4ad1-8680-3429095d5bef Content-Length: 0 Date: Wed, 30 May 2018 06:03:57 GMT Connection: keep-alive
RESP BODY: Omitted, Content-Type is set to text/plain; charset=utf8. Only application/json responses have their bodies logged.
DEBUG (session:395) RESP: [302] Content-Type: text/plain; charset=utf8 Location: http://controller:8774/v2.1/ X-Compute-Request-Id: req-4ff0c878-32a4-4ad1-8680-3429095d5bef Content-Length: 0 Date: Wed, 30 May 2018 06:03:57 GMT Connection: keep-alive
RESP BODY: Omitted, Content-Type is set to text/plain; charset=utf8. Only application/json responses have their bodies logged.
DEBUG (connectionpool:401) "GET /v2.1/ HTTP/1.1" 200 384
RESP: [200] Content-Length: 384 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-ffa048e6-2bcb-4bef-991a-1d7f8df48dd9 Date: Wed, 30 May 2018 06:03:57 GMT Connection: keep-alive
RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://controller:8774/v2.1/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.42", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}
DEBUG (session:395) RESP: [200] Content-Length: 384 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Compute-Request-Id: req-ffa048e6-2bcb-4bef-991a-1d7f8df48dd9 Date: Wed, 30 May 2018 06:03:57 GMT Connection: keep-alive
RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "http://controller:8774/v2.1/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.42", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}
GET call to compute for http://controller:8774/v2.1/ used request id req-ffa048e6-2bcb-4bef-991a-1d7f8df48dd9
DEBUG (session:640) GET call to compute for http://controller:8774/v2.1/ used request id req-ffa048e6-2bcb-4bef-991a-1d7f8df48dd9
DEBUG (extension:180) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:180) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:180) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:180) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:180) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:180) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (session:347) REQ: curl -g -i -X GET http://controller:8774/v2.1/servers/detail -H "OpenStack-API-Version: compute 2.41" -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.41" -H "X-Auth-Token: {SHA1}f7a1d3aa256bf97008b37c21c1b4b702bac62158"
DEBUG (connectionpool:401) "GET /v2.1/servers/detail HTTP/1.1" 500 208
DEBUG (session:395) RESP: [500] Openstack-Api-Version: compute 2.41 X-Openstack-Nova-Api-Version: 2.41 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 208 X-Compute-Request-Id: req-404f0ba7-42f0-4059-904c-7a272156774b Date: Wed, 30 May 2018 06:03:57 GMT Connection: keep-alive
RESP BODY: {"computeFault": {"message": "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.\n<class 'nova.exception.OrphanedObjectError'>", "code": 500}}
DEBUG (session:640) GET call to compute for http://controller:8774/v2.1/servers/detail used request id req-404f0ba7-42f0-4059-904c-7a272156774b
DEBUG (shell:959) Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.OrphanedObjectError'> (HTTP 500) (Request-ID: req-404f0ba7-42f0-4059-904c-7a272156774b)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 957, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 879, in main
args.func(self.cs, args)
File "/usr/lib/python2.7/site-packages/novaclient/v2/shell.py", line 1710, in do_list
limit=args.limit)
File "/usr/lib/python2.7/site-packages/novaclient/v2/servers.py", line 861, in list
"servers")
File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 254, in _list
resp, body = self.api.client.get(url)
File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 80, in request
raise exceptions.from_response(resp, body, url, method)
ClientException: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.OrphanedObjectError'> (HTTP 500) (Request-ID: req-404f0ba7-42f0-4059-904c-7a272156774b)
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception.OrphanedObjectError'> (HTTP 500) (Request-ID: req-404f0ba7-42f0-4059-904c-7a272156774b)
Could you provide a bit more details:
* what are the steps that led to the failing nova list command? Did you created servers with nova boot or did any other operations on the existing servers (e.g. migrate / evacuate / resize /rebuild/ shelve)?
* please provide the whole stack trace from the nova-api log?