[nova] Instance is not shutoff after nova-compute start for a long time

[Sergey Arkhipov]

Instance is not shutoff after nova-compute start for a long time.

Steps to reproduce:
    * Choose instance and detect compute which manages it
    * SSH to compute and stop nova-compute with 'service nova-compute stop'
    * Wait for 2-3 minutes
    * Proceed to Horizon and click 'Shutoff instance'
    * Check that no error is shown, instance has not changed its state
    * Wait for 1 minute, proceed to the nova host again (where 'nova-compute' is not running)
    * Start nova-compute with 'service nova-compute start'
    * Wait for 5 minutes, check that instance has not changed its state and still accessible using webconsole at least.
    * Try to shutoff it again, with working nova-compute

Expected result:
    One of 2 results were expected:
        * Warning or error message that it is not possible complete operation right now
        * Instance is shut after nova-compute is up and running again

Real result:
    * No error in Horizon
    * Instance is still running and accessible

I've checked with instance '39d5f023-4103-4e11-9ba5-ca86d1803453', please check attached logs.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "199"
  build_id: "199"
  fuel-nailgun_sha: "a65871a955dd0886b9c2e18f7a24381f10753136"
  python-fuelclient_sha: "3e7738fd3fb18a2d5f53b1ecc9706dc53b65a511"
  fuel-agent_sha: "88c993eab990ddb7b5f038490295f52c602aa58b"
  fuel-nailgun-agent_sha: "b56f832abc18aee9a8c603fd6cc2055c5f4287bc"
  astute_sha: "c8400f51b0b92254da206de55ef89d17fdf35393"
  fuel-library_sha: "33c0fa3aada734dc9e6f315197ce0e4a16f5987c"
  fuel-ostf_sha: "11afd5743a12b1006317d3ca7000d1ede77bdae2"
  fuel-createmirror_sha: "994fed9b1ed889718b61a59733275c08c2dd4c64"
  fuelmenu_sha: "d12061b1aee82f81b3d074de74ea27a6e962a686"
  shotgun_sha: "c377d163519f6d10b69a654019d6086ba5f14edc"
  network-checker_sha: "2c62cd52655ea6456ff6294fd63f18d6ea54fe38"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "22fe551f5525d11a1854fd87dbc8c77fae8fec08"

[Roman Podoliaka (rpodolyaka)]

I checked this both on env #307 and in the code:

The shutoff action is sent as an RPC CAST to nova-compute. Even if a particular nova-compute service is down at the moment of time, the message will be stored in the message queue until nova-compute is up again, and then be consumed properly, shutting the instance down.

The status of instance is updated properly in the DB (i.e. nova list will show it as SHUTOFF) and there is no qemu process running on the compute node.

[Roman Podoliaka (rpodolyaka)]

Sergey, could you please give it another try?

[Roman Podoliaka (rpodolyaka)]

No new repro or feedback in a month, closing as Invalid.