========================================================================================
$ virsh --version
1.2.16
Nova version: mitaka
1. verify the status of the instance is "ACTIVE":
$ nova list --all-tenants --fields name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
| 406b7bfb-aa47-4b71-bd2d-5bf6347ac9ef | boot_from_cinder | ACTIVE | active | nova | ardana-cp1-comp0001-mgmt |
| a5b5de2f-a35a-471c-a895-e5e3942b4b4e | test | ACTIVE | active | nova | ardana-cp1-comp0001-mgmt |
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
2. verify Cinder volume is attached:
$ nova show a5b5de2f-a35a-471c-a895-e5e3942b4b4e | egrep "Property|os-extended-volumes|image"
| Property | Value |
| image | Attempt to boot from volume - no image supplied |
| os-extended-volumes:volumes_attached | [{"id": "b0dd17d9-cae9-44ff-a6f6-6cacef785a60"}] |
$ cinder list | egrep "b0dd17d9-cae9-44ff-a6f6-6cacef785a60"
| b0dd17d9-cae9-44ff-a6f6-6cacef785a60 | in-use | | 1 | - | true | a5b5de2f-a35a-471c-a895-e5e3942b4b4e |
$ nova availability-zone-list | egrep -A 10 "^\| nova"
| nova | available |
| |- ardana-cp1-comp0001-mgmt | |
| | |- nova-compute | enabled :-) 2018-09-19T02:56:19.000000 |
| |- ardana-cp1-comp0002-mgmt | |
| | |- nova-compute | enabled :-) 2018-09-19T02:56:21.000000 |
3. stop nova-compute service on the compute node the instance is currently running:
$ sudo service nova-compute stop
4. verify nova-compute Status is "down":
$ nova service-list | egrep "Host|nova-compute"
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
| 41 | nova-compute | ardana-cp1-comp0002-mgmt | nova | enabled | up | 2018-09-19T03:24:51.000000 | - |
| 43 | nova-compute | ardana-cp1-comp0001-mgmt | nova | enabled | down | 2018-09-19T03:23:50.000000 | - |
5. evacuate Nova instance to target-node:
$ nova --debug evacuate --on-shared-storage a5b5de2f-a35a-471c-a895-e5e3942b4b4e ardana-cp1-comp0002-mgmt
DEBUG (extension:180) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (extension:180) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:180) found extension EntryPoint.parse('noauth = cinderclient.contrib.noauth:CinderNoAuthLoader')
DEBUG (extension:180) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:180) found extension EntryPoint.parse('none = keystoneauth1.loading._plugins.noauth:NoAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:180) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:180) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3adfspassword = keystoneauth1.extras._saml2._loading:ADFSPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:180) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:180) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:180) found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')
DEBUG (extension:180) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')
DEBUG (session:375) REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:5000/v3 -H "Accept: application/json" -H "User-Agent: nova keystoneauth1/3.1.0 python-requests/2.18.2 CPython/2.7.13"
DEBUG (connectionpool:824) Starting new HTTPS connection (1): 10.41.180.1
DEBUG (connectionpool:396) https://10.41.180.1:5000 "GET /v3 HTTP/1.1" 200 251
DEBUG (session:423) RESP: [200] Date: Wed, 19 Sep 2018 03:26:31 GMT Server: Apache Content-Length: 251 Vary: X-Auth-Token x-openstack-request-id: req-e1d52b29-ae0b-47c1-8c20-f2f8157120a1 Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2017-02-22T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.8", "links": [{"href": "https://10.41.180.1:5000/v3/", "rel": "self"}]}}
DEBUG (session:727) GET call to None for https://10.41.180.1:5000/v3 used request id req-e1d52b29-ae0b-47c1-8c20-f2f8157120a1
DEBUG (base:165) Making authentication request to https://10.41.180.1:5000/v3/auth/tokens
DEBUG (connectionpool:396) https://10.41.180.1:5000 "POST /v3/auth/tokens HTTP/1.1" 201 12340
DEBUG (base:170) {"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5af93195cd2344b98d238697a97529f1", "name": "key-manager:service-admin"}, {"id": "c7fbf4afc6c4431f8b081cd34fb6de93", "name": "monasca-user"}, {"id": "b1e6fd11640b4df8a69fd34b94541796", "name": "key-manager:admin"}, {"id": "8e8b0345a3e648c58b981e72a4ae7f67", "name": "admin"}], "is_admin_project": false, "project": {"domain": {"id": "default", "name": "Default"}, "id": "01e8a0b8203a452d8e4d286b2da5fae1", "name": "admin"}, "catalog": [{"endpoints": [{"url": "https://10.41.180.1:9696/", "interface": "internal", "region": "region1", "region_id": "region1", "id": "765df6b1782540b5910fa0464ae0a8ed"}, {"url": "https://10.41.182.2:9696/", "interface": "public", "region": "region1", "region_id": "region1", "id": "9522172c2f1742c4b75a9f5553c03377"}, {"url": "https://10.41.180.1:9696/", "interface": "admin", "region": "region1", "region_id": "region1", "id": "d68c2b36d1fb429dbf2853fc67b76668"}], "type": "network", "id": "10349638ebb9459baa8e0006f5c13306", "name": "neutron"}, {"endpoints": [{"url": "https://10.41.180.1:8004/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "550be6be6a3345d6b2506b948cd0dd73"}, {"url": "https://10.41.182.2:8004/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "853a221df79d41c6aa4ff0fe9361b240"}, {"url": "https://10.41.180.1:8004/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "ea1a3170809943cda0a6ce3f465158c0"}], "type": "orchestration", "id": "155c777887dd4ecda4dacba1a8847e4c", "name": "heat"}, {"endpoints": [{"url": "https://10.41.182.2:8776/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "56e05fe7ffab4febb0a790e8e6cbbdde"}, {"url": "https://10.41.180.1:8776/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "97932a3de8954639be1e42ff2b0fb0c6"}, {"url": "https://10.41.180.1:8776/v1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "f96579aeda0842259c2dd8557c8a4f99"}], "type": "volume", "id": "2772c6557b6b44ae94bb4035cc093e91", "name": "cinder"}, {"endpoints": [{"url": "https://10.41.182.2:443", "interface": "public", "region": "region1", "region_id": "region1", "id": "2cb6dd69fd1c4f9abc20afdb2404176c"}, {"url": "https://10.41.180.1:443", "interface": "admin", "region": "region1", "region_id": "region1", "id": "e43a4fa58e9c4f3e8d0598e14bab7ea1"}, {"url": "https://10.41.180.1:443", "interface": "internal", "region": "region1", "region_id": "region1", "id": "ea07084251044b12b3527618913bb6b1"}], "type": "dashboard", "id": "312b66224c1446b19d3d3f6d89df5a73", "name": "horizon"}, {"endpoints": [{"url": "https://10.41.182.2:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "4e04c2476d94485d9de6b557ab59d11f"}, {"url": "https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "850b41b937e6476cb9cf8c4190653a0d"}, {"url": "https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "8f581d8e058b4c9d81c6ce061153e0b5"}], "type": "compute", "id": "32b0d4e3e36c49058ff500cc18d2c22d", "name": "nova"}, {"endpoints": [{"url": "https://10.41.180.1:8776/v2/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "14dc40e9931647ee8c7c89039a4d6a1c"}, {"url": "https://10.41.180.1:8776/v2/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "3011cf2a9f4e4c44ba338c3ad0893c2f"}, {"url": "https://10.41.182.2:8776/v2/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "773fde6921b44accbb0a71b21629ad42"}], "type": "volumev2", "id": "32cfb3cdc0f84e60b064e1398ce7c70b", "name": "cinderv2"}, {"endpoints": [{"url": "https://10.41.180.1:9292", "interface": "admin", "region": "region1", "region_id": "region1", "id": "5950ed7f3a084d188e5e9d0349967dfc"}, {"url": "https://10.41.182.2:9292", "interface": "public", "region": "region1", "region_id": "region1", "id": "da45fd45381849d19e007111442d2287"}, {"url": "https://10.41.180.1:9292", "interface": "internal", "region": "region1", "region_id": "region1", "id": "ff97715186ed4df89e10b732095e94ca"}], "type": "image", "id": "3827131e28ee4b2e9526e31b58f647d1", "name": "glance"}, {"endpoints": [{"url": "https://10.41.180.1:9001", "interface": "admin", "region": "region1", "region_id": "region1", "id": "a153fa317b114c27ab85ef79b30c1fdf"}, {"url": "https://10.41.182.2:9001", "interface": "public", "region": "region1", "region_id": "region1", "id": "ce66bf3d602344eaa151158f676a45d9"}, {"url": "https://10.41.180.1:9001", "interface": "internal", "region": "region1", "region_id": "region1", "id": "dd9c1e1657204e2d815903102141973a"}], "type": "dns", "id": "388153f31a044ea783a0195b0c2ea82e", "name": "designate"}, {"endpoints": [{"url": "https://10.41.180.1:5607/v3.0", "interface": "internal", "region": "region1", "region_id": "region1", "id": "009df1fd4e824c83a0b06e435d4bd306"}, {"url": "https://10.41.182.2:5607/v3.0", "interface": "public", "region": "region1", "region_id": "region1", "id": "526911dff0724260bdb3c7a5c9b6b047"}, {"url": "https://10.41.180.1:5607/v3.0", "interface": "admin", "region": "region1", "region_id": "region1", "id": "d1f5c04e13b2414a901d7492cb637eac"}], "type": "logging", "id": "3a68b23b34f448f485de9a5fd79c800e", "name": "kronos"}, {"endpoints": [{"url": "https://10.41.180.1:9095/api/v1/", "interface": "internal", "region": "region1", "region_id": "region1", "id": "7bafb829ab82491f881c9f7a4a1115e3"}, {"url": "https://10.41.180.1:9095/api/v1/", "interface": "admin", "region": "region1", "region_id": "region1", "id": "a1654c94a85145bda2bdf5f180d7a35c"}, {"url": "https://10.41.182.2:9095/api/v1/", "interface": "public", "region": "region1", "region_id": "region1", "id": "b46bcc5301474b04bec0a9165827e304"}], "type": "opsconsole", "id": "44c538e29c6c465e969c9bc5f084626a", "name": "opsconsole"}, {"endpoints": [{"url": "https://10.41.180.1:8070/v2.0", "interface": "admin", "region": "region1", "region_id": "region1", "id": "506c9b2fb4d247d5abc72773f77a1716"}, {"url": "https://10.41.180.1:8070/v2.0", "interface": "internal", "region": "region1", "region_id": "region1", "id": "a6779864864044ea84ada35310cf567b"}, {"url": "https://10.41.182.2:8070/v2.0", "interface": "public", "region": "region1", "region_id": "region1", "id": "e3a531bcfd1a496ea9b03f338e1ca790"}], "type": "monitoring", "id": "5e660bea43da46cdabba94fe4f4473d7", "name": "monasca"}, {"endpoints": [{"url": "https://10.41.180.1:9085", "interface": "admin", "region": "region1", "region_id": "region1", "id": "732a2afed0e64ba3ba9abbcbb74de5b3"}, {"url": "https://10.41.182.2:9085", "interface": "public", "region": "region1", "region_id": "region1", "id": "7f6a6058e7c3481faae0918c1e0647e6"}, {"url": "https://10.41.180.1:9085", "interface": "internal", "region": "region1", "region_id": "region1", "id": "9d68f6c60db34a1da537d67fe5021a97"}], "type": "lifecycle", "id": "6c45e1ffacb74c49a50f7f68e9fe8512", "name": "ardana"}, {"endpoints": [{"url": "https://10.41.180.1:8776/v3/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "097554bd11d2471c878c533758486461"}, {"url": "https://10.41.182.2:8776/v3/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "854f7fc597914eb3b37c9c5538ae1b1a"}, {"url": "https://10.41.180.1:8776/v3/01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "ccff900043164bd7b5db077db2a52517"}], "type": "volumev3", "id": "7bfb7bda334946ed94461de15869353f", "name": "cinderv3"}, {"endpoints": [{"url": "https://10.41.180.1:5000/", "interface": "internal", "region": "region1", "region_id": "region1", "id": "51e43a807dfb492dacdff041f591243a"}, {"url": "https://10.41.182.2:5000/", "interface": "public", "region": "region1", "region_id": "region1", "id": "cf9dabcf426d4f7689accd79d20d9c59"}, {"url": "https://10.41.180.1:35357/", "interface": "admin", "region": "region1", "region_id": "region1", "id": "ff2dfc994ef4436aacb412f9b0cabbf9"}], "type": "identity", "id": "91127023335c48288b4b7feb28ebc0d8", "name": "keystone"}, {"endpoints": [{"url": "https://10.41.182.2:8080/v1/AUTH_01e8a0b8203a452d8e4d286b2da5fae1", "interface": "public", "region": "region1", "region_id": "region1", "id": "5d0caaf83928477c8b7f3e7a37d31199"}, {"url": "https://10.41.180.1:8080/v1/AUTH_01e8a0b8203a452d8e4d286b2da5fae1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "6fd718414ff049ae9d5e5df6c40ecd9b"}, {"url": "https://10.41.180.1:8080/v1/AUTH_01e8a0b8203a452d8e4d286b2da5fae1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "e4ea65965691460383f86bc3ddac6ec9"}], "type": "object-store", "id": "a74f12f2f259409ea8333cab152d5977", "name": "swift"}, {"endpoints": [{"url": "https://10.41.182.2:9511/v1", "interface": "public", "region": "region1", "region_id": "region1", "id": "06404af8e25f4dbb9bbab2eee3e1c926"}, {"url": "https://10.41.180.1:9511/v1", "interface": "admin", "region": "region1", "region_id": "region1", "id": "493fda65cb704db3a6da8be61a4e342d"}, {"url": "https://10.41.180.1:9511/v1", "interface": "internal", "region": "region1", "region_id": "region1", "id": "c601a375fb1840e5a3763022d04f1587"}], "type": "container-infra", "id": "a99310461f38491b8e0856447553749a", "name": "magnum"}, {"endpoints": [{"url": "https://10.41.180.1:8777/", "interface": "internal", "region": "region1", "region_id": "region1", "id": "1eb183923a1a4e8fb1dbad18739e2ade"}, {"url": "https://10.41.180.1:8777/", "interface": "admin", "region": "region1", "region_id": "region1", "id": "e33657947ae342f7ac8b6bc08d7f9204"}, {"url": "https://10.41.182.2:8777/", "interface": "public", "region": "region1", "region_id": "region1", "id": "e449dba6beee494482128a558eb961af"}], "type": "metering", "id": "ac511da6fc7243989950d6c19b3fe4c6", "name": "ceilometer"}, {"endpoints": [{"url": "https://10.41.182.2:9311", "interface": "public", "region": "region1", "region_id": "region1", "id": "047935786a904843be72beaf87494e42"}, {"url": "https://10.41.180.1:9311", "interface": "internal", "region": "region1", "region_id": "region1", "id": "3a88a0f75a1f45f98135bae5a475ff8e"}, {"url": "https://10.41.180.1:9311", "interface": "admin", "region": "region1", "region_id": "region1", "id": "98ac9f0e017d4b3b9fe5060b84f6af69"}], "type": "key-manager", "id": "b177046b16c94a589f71972ec2e14e20", "name": "barbican"}, {"endpoints": [{"url": "https://10.41.180.1:9090", "interface": "internal", "region": "region1", "region_id": "region1", "id": "00461614423e459d9f33a983af2dbf13"}, {"url": "https://10.41.180.1:9090", "interface": "admin", "region": "region1", "region_id": "region1", "id": "ec0ef92c1403423ba7c8e7a620868b9f"}, {"url": "https://10.41.182.2:9090", "interface": "public", "region": "region1", "region_id": "region1", "id": "ec1747cf6e784b34be0625305349dc21"}], "type": "backup", "id": "bfd3366e0753441ca9104411e116751d", "name": "freezer"}, {"endpoints": [{"url": "https://10.41.182.2:8780/", "interface": "public", "region": "region1", "region_id": "region1", "id": "e917cf38ad3846d1b1b506db4ba092ae"}, {"url": "https://10.41.180.1:8780/", "interface": "internal", "region": "region1", "region_id": "region1", "id": "f09e28bb1ee34426b04b379907817eaa"}, {"url": "https://10.41.180.1:8780/", "interface": "admin", "region": "region1", "region_id": "region1", "id": "fe11364832f94bae921ba699c812a640"}], "type": "placement", "id": "dae1b03b95a543a2b885a4595a5416b3", "name": "placement"}], "expires_at": "2018-09-19T07:26:32.000000Z", "audit_ids": ["r8GvcA2NQku39FgoHokP_A"], "issued_at": "2018-09-19T03:26:32.000000Z", "user": {"domain": {"id": "default", "name": "Default"}, "password_expires_at": null, "name": "admin", "id": "00ecab1fe1684e0f8369dc077ae1baf2"}}}
REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4"
DEBUG (session:375) REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1 -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4"
DEBUG (connectionpool:824) Starting new HTTPS connection (1): 10.41.180.1
DEBUG (connectionpool:396) https://10.41.180.1:8774 "GET /v2.1/01e8a0b8203a452d8e4d286b2da5fae1 HTTP/1.1" 404 112
RESP: [404] Content-Length: 112 Content-Type: application/json X-Openstack-Request-Id: req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e X-Compute-Request-Id: req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e Date: Wed, 19 Sep 2018 03:26:32 GMT Connection: keep-alive
RESP BODY: {"message": "The resource could not be found.
\n\n\n", "code": "404 Not Found", "title": "Not Found"}
DEBUG (session:423) RESP: [404] Content-Length: 112 Content-Type: application/json X-Openstack-Request-Id: req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e X-Compute-Request-Id: req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e Date: Wed, 19 Sep 2018 03:26:32 GMT Connection: keep-alive
RESP BODY: {"message": "The resource could not be found.
\n\n\n", "code": "404 Not Found", "title": "Not Found"}
GET call to compute for https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1 used request id req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e
DEBUG (session:727) GET call to compute for https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1 used request id req-d8a767da-1e5f-47b8-b5b4-fed04ebdc58e
REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4"
DEBUG (session:375) REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:8774/v2.1/ -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4"
DEBUG (connectionpool:396) https://10.41.180.1:8774 "GET /v2.1/ HTTP/1.1" 200 386
RESP: [200] Content-Length: 386 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Openstack-Request-Id: req-37f70e2b-2c1a-4996-95da-6deda8b1845b X-Compute-Request-Id: req-37f70e2b-2c1a-4996-95da-6deda8b1845b Date: Wed, 19 Sep 2018 03:26:32 GMT Connection: keep-alive
RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "https://10.41.182.2:8774/v2.1/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.53", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}
DEBUG (session:423) RESP: [200] Content-Length: 386 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Openstack-Request-Id: req-37f70e2b-2c1a-4996-95da-6deda8b1845b X-Compute-Request-Id: req-37f70e2b-2c1a-4996-95da-6deda8b1845b Date: Wed, 19 Sep 2018 03:26:32 GMT Connection: keep-alive
RESP BODY: {"version": {"status": "CURRENT", "updated": "2013-07-23T11:33:21Z", "links": [{"href": "https://10.41.182.2:8774/v2.1/", "rel": "self"}, {"href": "http://docs.openstack.org/", "type": "text/html", "rel": "describedby"}], "min_version": "2.1", "version": "2.53", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.compute+json;version=2.1"}], "id": "v2.1"}}
GET call to compute for https://10.41.180.1:8774/v2.1/ used request id req-37f70e2b-2c1a-4996-95da-6deda8b1845b
DEBUG (session:727) GET call to compute for https://10.41.180.1:8774/v2.1/ used request id req-37f70e2b-2c1a-4996-95da-6deda8b1845b
DEBUG (extension:180) found extension EntryPoint.parse('v1password = swiftclient.authv1:PasswordLoader')
DEBUG (extension:180) found extension EntryPoint.parse('token_endpoint = openstackclient.api.auth_plugin:TokenEndpoint')
DEBUG (extension:180) found extension EntryPoint.parse('noauth = cinderclient.contrib.noauth:CinderNoAuthLoader')
DEBUG (extension:180) found extension EntryPoint.parse('v2token = keystoneauth1.loading._plugins.identity.v2:Token')
DEBUG (extension:180) found extension EntryPoint.parse('none = keystoneauth1.loading._plugins.noauth:NoAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3oauth1 = keystoneauth1.extras.oauth1._loading:V3OAuth1')
DEBUG (extension:180) found extension EntryPoint.parse('admin_token = keystoneauth1.loading._plugins.admin_token:AdminToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcauthcode = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAuthorizationCode')
DEBUG (extension:180) found extension EntryPoint.parse('v2password = keystoneauth1.loading._plugins.identity.v2:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3samlpassword = keystoneauth1.extras._saml2._loading:Saml2Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3password = keystoneauth1.loading._plugins.identity.v3:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3adfspassword = keystoneauth1.extras._saml2._loading:ADFSPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcaccesstoken = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectAccessToken')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcpassword = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectPassword')
DEBUG (extension:180) found extension EntryPoint.parse('v3kerberos = keystoneauth1.extras.kerberos._loading:Kerberos')
DEBUG (extension:180) found extension EntryPoint.parse('token = keystoneauth1.loading._plugins.identity.generic:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3oidcclientcredentials = keystoneauth1.loading._plugins.identity.v3:OpenIDConnectClientCredentials')
DEBUG (extension:180) found extension EntryPoint.parse('v3tokenlessauth = keystoneauth1.loading._plugins.identity.v3:TokenlessAuth')
DEBUG (extension:180) found extension EntryPoint.parse('v3token = keystoneauth1.loading._plugins.identity.v3:Token')
DEBUG (extension:180) found extension EntryPoint.parse('v3totp = keystoneauth1.loading._plugins.identity.v3:TOTP')
DEBUG (extension:180) found extension EntryPoint.parse('password = keystoneauth1.loading._plugins.identity.generic:Password')
DEBUG (extension:180) found extension EntryPoint.parse('v3fedkerb = keystoneauth1.extras.kerberos._loading:MappedKerberos')
DEBUG (extension:180) found extension EntryPoint.parse('gnocchi-basic = gnocchiclient.auth:GnocchiBasicLoader')
DEBUG (extension:180) found extension EntryPoint.parse('gnocchi-noauth = gnocchiclient.auth:GnocchiNoAuthLoader')
DEBUG (session:375) REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X GET https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e -H "User-Agent: python-novaclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4"
DEBUG (connectionpool:396) https://10.41.180.1:8774 "GET /v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e HTTP/1.1" 200 1564
DEBUG (session:423) RESP: [200] Content-Length: 1564 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Openstack-Request-Id: req-34ed0ada-2778-450d-80cc-1432e5907244 X-Compute-Request-Id: req-34ed0ada-2778-450d-80cc-1432e5907244 Date: Wed, 19 Sep 2018 03:26:33 GMT Connection: keep-alive
RESP BODY: {"server": {"OS-EXT-STS:task_state": null, "addresses": {"Internal": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:12:3e:1e", "version": 4, "addr": "192.168.1.9", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "https://10.41.182.2:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e", "rel": "self"}, {"href": "https://10.41.182.2:8774/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e", "rel": "bookmark"}], "image": "", "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-00000003", "OS-SRV-USG:launched_at": "2018-09-14T08:25:10.000000", "flavor": {"id": "1", "links": [{"href": "https://10.41.182.2:8774/01e8a0b8203a452d8e4d286b2da5fae1/flavors/1", "rel": "bookmark"}]}, "id": "a5b5de2f-a35a-471c-a895-e5e3942b4b4e", "security_groups": [{"name": "default"}], "user_id": "00ecab1fe1684e0f8369dc077ae1baf2", "OS-DCF:diskConfig": "AUTO", "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "status": "ACTIVE", "updated": "2018-09-19T02:51:40Z", "hostId": "d9aaee93bd126da68e7e77fb2937616af4e51b53e27774fc199a021d", "OS-EXT-SRV-ATTR:host": "ardana-cp1-comp0001-mgmt", "OS-SRV-USG:terminated_at": null, "key_name": null, "OS-EXT-SRV-ATTR:hypervisor_hostname": "ardana-cp1-comp0001-mgmt", "name": "test", "created": "2018-09-14T08:24:05Z", "tenant_id": "01e8a0b8203a452d8e4d286b2da5fae1", "os-extended-volumes:volumes_attached": [{"id": "b0dd17d9-cae9-44ff-a6f6-6cacef785a60"}], "metadata": {}}}
DEBUG (session:727) GET call to compute for https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e used request id req-34ed0ada-2778-450d-80cc-1432e5907244
DEBUG (session:375) REQ: curl -g -i --cacert "/etc/ssl/ca-bundle.pem" -X POST https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e/action -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}a6e223063d6a36c9176a7f8a5b6f63271c0068d4" -d '{"evacuate": {"host": "ardana-cp1-comp0002-mgmt", "onSharedStorage": true}}'
DEBUG (connectionpool:396) https://10.41.180.1:8774 "POST /v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e/action HTTP/1.1" 200 19
DEBUG (session:423) RESP: [200] Content-Length: 19 Content-Type: application/json Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version, X-OpenStack-Nova-API-Version X-Openstack-Request-Id: req-01f9d88c-8f91-4f4a-bb67-2e478c10581e X-Compute-Request-Id: req-01f9d88c-8f91-4f4a-bb67-2e478c10581e Date: Wed, 19 Sep 2018 03:26:33 GMT Connection: keep-alive
RESP BODY: {"adminPass": null}
DEBUG (session:727) POST call to compute for https://10.41.180.1:8774/v2.1/01e8a0b8203a452d8e4d286b2da5fae1/servers/a5b5de2f-a35a-471c-a895-e5e3942b4b4e/action used request id req-01f9d88c-8f91-4f4a-bb67-2e478c10581e
+-----------+-------+
| Property | Value |
+-----------+-------+
| adminPass | - |
+-----------+-------+
# nova list --field name,status,OS-EXT-STS:vm_state,OS-EXT-AZ:availability_zone,OS-EXT-SRV-ATTR:hypervisor_hostname
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
| ID | Name | Status | OS-EXT-STS: Vm State | OS-EXT-AZ: Availability Zone | OS-EXT-SRV-ATTR: Hypervisor Hostname |
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
| a5b5de2f-a35a-471c-a895-e5e3942b4b4e | test | ERROR | error | nova | ardana-cp1-comp0001-mgmt |
+--------------------------------------+------------------+--------+----------------------+------------------------------+--------------------------------------+
Nova compute log file has
Setting instance vm_state to ERROR: TypeError: 'NoneType' object is not iterable
Traceback (most recent call last):
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 6920, in _error_out_instance_on_exception
yield
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 2842, in rebuild_instance
bdms, recreate, on_shared_storage, preserve_ephemeral)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 2890, in _do_rebuild_instance_with_claim
self._do_rebuild_instance(*args, **kwargs)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 3012, in _do_rebuild_instance
self._rebuild_default_impl(**kwargs)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 2731, in _rebuild_default_impl
detach_block_devices(context, bdms)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 2989, in detach_block_devices
destroy_bdm=False)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 5010, in _detach_volume
self._notify_volume_usage_detach(context, instance, bdm)
File "/opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/
compute/manager.py", line 4975, in _notify_volume_usage_detach
rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats
TypeError: 'NoneType' object is not iterable\]
Edited lib/python2.7/site-packages/nova/compute/manager.py and added a try/except
$ sudo vi manager.py
...
def _notify_volume_usage_detach(self, context, instance, bdm):
if CONF.volume_usage_poll_interval <= 0:
return
vol_stats = []
mp = bdm.device_name
# Handle bootable volumes which will not contain /dev/
if '/dev/' in mp:
mp = mp[5:]
try:
LOG.info("Device info: %s" % mp)
vol_stats = self.driver.block_stats(instance, mp)
except NotImplementedError:
return
LOG.debug("Updating volume usage cache with totals", instance=instance)
-- rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats
++ try:
++ rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats
++ except TypeError:
++ rd_req = 0
++ rd_bytes = 0
++ wr_req = 0
++ wr_bytes = 0
++ flush_ops = 0
vol_usage = objects.VolumeUsage(context)
vol_usage.volume_id = bdm.volume_id
vol_usage.instance_uuid = instance.uuid
vol_usage.project_id = instance.project_id
vol_usage.user_id = instance.user_id
vol_usage.availability_zone = instance.availability_zone
Evacuate now works correctly
1. on both compute nodes stop "nova-compute" systemd service:
2. modify /opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/compute/manager.py
see also "Case 1" from https://bugzilla.suse.com/show_bug.cgi?id=1106361#c28
### amach rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats
try:
rd_req, rd_bytes, wr_req, wr_bytes, flush_ops = vol_stats
except TypeError:
rd_req = 0
rd_bytes = 0
wr_req = 0
wr_bytes = 0
flush_ops = 0
3. start "nova-compute" systemd services on both nodes again
4. Stop "nova-compute" service on "compute1 / ardana-cp1-comp0001-mgmt" and verify current status of instance volume etc.:
ardana@ardana-cp1-c0-m1-mgmt:~> nova service-list | grep comp000
| 55 | nova-compute | ardana-cp1-comp0002-mgmt | nova | enabled | up | 2018-09-27T11:19:56.000000 | - |
| 58 | nova-compute | ardana-cp1-comp0001-mgmt | nova | enabled | up | 2018-09-27T11:19:56.000000 | - |
ardana-cp1-comp0001-mgmt:~ # systemctl stop nova-compute
ardana-cp1-comp0001-mgmt:~ # virsh list
Id Name State
----------------------------------------------------
3 instance-00000004 running
ardana@ardana-cp1-c0-m1-mgmt:~> nova service-list grep comp000
| 58 | nova-compute | ardana-cp1-comp0001-mgmt | nova | enabled | down | 2018-09-27T11:21:16.000000 | - |
ardana@ardana-cp1-c0-m1-mgmt:~> openstack server show c1e43aae-e376-44d5-9fa6-81b4db6deb4f
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ardana-cp1-comp0001-mgmt |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ardana-cp1-comp0001-mgmt |
| OS-EXT-SRV-ATTR:instance_name | instance-00000004 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2018-09-27T11:15:31.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | demo-network1=192.168.1.7 |
| config_drive | |
| created | 2018-09-26T16:11:17Z |
| flavor | m1.tiny (1) |
| hostId | e49c6c362473497461a915d600ed2490d72cae188241c7f5cc82aeef |
| id | c1e43aae-e376-44d5-9fa6-81b4db6deb4f |
| image | |
| key_name | admin-key |
| name | demo-instance |
| progress | 0 |
| project_id | 903a4f59ade74c9baee9d97cebf4c9c9 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2018-09-27T11:15:32Z |
| user_id | ab53ae38f8344714ae61877cea44a4a0 |
| volumes_attached | id='44d42373-3375-4338-a481-6a736f49c473' |
+-------------------------------------+----------------------------------------------------------+
ardana@ardana-cp1-c0-m1-mgmt:~> openstack volume show 44d42373-3375-4338-a481-6a736f49c473 --fit-width
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'c1e43aae-e376-44d5-9fa6-81b4db6deb4f', u'attachment_id': u'bc873bec-745a-4f55-b93d-7ae8c75abba9', u'attached_at': u'2018-09-27T11:15:28.000000', u'host_name': None, u'volume_id': |
| | u'44d42373-3375-4338-a481-6a736f49c473', u'device': u'/dev/vda', u'id': u'44d42373-3375-4338-a481-6a736f49c473'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2018-09-26T15:51:26.000000 |
| description | None |
| encrypted | False |
| id | 44d42373-3375-4338-a481-6a736f49c473 |
| multiattach | False |
| name | demo-bootable_volume |
| os-vol-tenant-attr:tenant_id | 903a4f59ade74c9baee9d97cebf4c9c9 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | None |
| size | 5 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | None |
| updated_at | 2018-09-27T11:15:29.000000 |
| user_id | ab53ae38f8344714ae61877cea44a4a0 |
| volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'raw', u'image_name': u'cirros-0.4.0-raw', u'image_id': u'a3690c3e-1637-4eb7-9d1b-74c70fdd3c60', u'checksum': |
| | u'ba3cd24377dde5dfdd58728894004abb', u'min_disk': u'0', u'size': u'46137344'} |
+------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
6. evacuate instance to "compute2 / ardana-cp1-comp0002-mgmt"
ardana@ardana-cp1-c0-m1-mgmt:~> nova --debug evacuate --on-shared-storage c1e43aae-e376-44d5-9fa6-81b4db6deb4f ardana-cp1-comp0002-mgmt
openstack server show c1e43aae-e376-44d5-9fa6-81b4db6deb4f
+-------------------------------------+----------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | ardana-cp1-comp0002-mgmt |
| OS-EXT-SRV-ATTR:hypervisor_hostname | ardana-cp1-comp0002-mgmt |
| OS-EXT-SRV-ATTR:instance_name | instance-00000004 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2018-09-27T11:34:03.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | demo-network1=192.168.1.7 |
| config_drive | |
| created | 2018-09-26T16:11:17Z |
| flavor | m1.tiny (1) |
| hostId | 892bc1ae46bd28df0a68686a3dc6336c8571fced6cdc648fa596e604 |
| id | c1e43aae-e376-44d5-9fa6-81b4db6deb4f |
| image | |
| key_name | admin-key |
| name | demo-instance |
| progress | 0 |
| project_id | 903a4f59ade74c9baee9d97cebf4c9c9 |
| properties | |
| security_groups | name='default' |
| status | ACTIVE |
| updated | 2018-09-27T11:34:04Z |
| user_id | ab53ae38f8344714ae61877cea44a4a0 |
| volumes_attached | id='44d42373-3375-4338-a481-6a736f49c473' |
+-------------------------------------+----------------------------------------------------------+
nova-compute.log from compute2:
2018-09-27 11:34:03.581 7276 INFO nova.compute.manager [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] [instance: c1e43aae-e376-44d5-9fa6-81b4db6deb4f] VM Resumed (Lifecycle Event)
2018-09-27 11:34:03.583 7276 DEBUG oslo_messaging._drivers.amqpdriver [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] CALL msg_id: 03f241ba10004bedab9e58d2af2b67f8 exchange 'nova' topic 'conductor' _send /opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:550
2018-09-27 11:34:03.584 7276 TRACE oslo.messaging._drivers.impl_rabbit [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] Connection._publish: sending message {'oslo.message': '{"_context_domain": null, "_msg_id": "03f241ba10004bedab9e58d2af2b67f8", "_context_global_request_id": null, "_context_quota_class": null, "_context_read_only": false, "_context_request_id": "req-98a014f5-f41a-4ec9-9cc9-29ac2e87496d", "_context_service_catalog": [], "args": {"object_versions": {"PciDevicePoolList": "1.1", "ServiceList": "1.19", "PciDevicePool": "1.1", "Service": "1.22", "TagList": "1.1", "InstancePCIRequests": "1.1", "VirtCPUModel": "1.0", "MigrationContext": "1.1", "SecurityGroup": "1.2", "Instance": "2.3", "KeyPair": "1.4", "KeyPairList": "1.3", "DeviceMetadata": "1.0", "InstancePCIRequest": "1.1", "EC2Ids": "1.0", "InstanceInfoCache": "1.5", "VirtCPUFeature": "1.0", "Flavor": "1.1", "SecurityGroupList": "1.1", "PciDevice": "1.6", "VirtCPUTopology": "1.0", "InstanceNUMACell": "1.4", "DeviceBus": "1.0", "ComputeNode": "1.18", "InstanceFault": "1.2", "Tag": "1.1", "HVSpec": "1.2", "InstanceDeviceMetadata": "1.0", "InstanceNUMATopology": "1.3", "PciDeviceList": "1.3"}, "objmethod": "get_by_uuid", "args": ["c1e43aae-e376-44d5-9fa6-81b4db6deb4f"], "objname": "Instance", "kwargs": {"expected_attrs": []}}, "_unique_id": "4bac4f74e2714036bafe83d21ec101aa", "_context_resource_uuid": null, "_context_instance_lock_checked": false, "_context_is_admin_project": true, "_context_user": null, "_context_user_id": null, "_context_project_name": null, "_context_read_deleted": "yes", "_context_user_identity": "- - - - -", "_reply_q": "reply_de6efc1a08d849f082e816cb9ba2621e", "_context_auth_token": null, "_context_show_deleted": false, "_context_tenant": null, "_context_roles": [], "_context_is_admin": true, "version": "3.0", "_context_project_id": null, "_context_project_domain": null, "_context_timestamp": "2018-09-27T11:34:03.582688", "_context_user_domain": null, "_context_user_name": null, "method": "object_class_action_versions", "_context_remote_address": null}', 'oslo.version': '2.0'} to Exchange nova(topic) with routing key conductor
2018-09-27 11:34:03.586 7276 DEBUG nova.virt.libvirt.driver [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] [instance: c1e43aae-e376-44d5-9fa6-81b4db6deb4f] Instance is running spawn /opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:2906
2018-09-27 11:34:03.589 7276 INFO nova.virt.libvirt.driver [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] [instance: c1e43aae-e376-44d5-9fa6-81b4db6deb4f] Instance spawned successfully.
2018-09-27 11:34:03.590 7276 DEBUG nova.compute.manager [req-7cad8e6e-1603-4e9e-a237-1b1c5211b507 1cdb21330345486b8826d4108e2e1a95 e4d99dc428ac405689099fac1518fae3] [instance: c1e43aae-e376-44d5-9fa6-81b4db6deb4f] Checking state _get_power_state /opt/stack/venv/nova-20180528T092954Z/lib/python2.7/site-packages/nova/compute/manager.py:1187