cinder service-disable should not change current state of cinder serivces

Bug #1397167 reported by Peter Wang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
harsh mishra

Bug Description

some service down as below:
stack@ubuntu-server7:~/devstack$ cinder service-list
+------------------+---------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+---------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | ubuntu-server7 | nova | enabled | up | 2014-11-27T22:16:49.000000 | None |
| cinder-volume | ubuntu-server7@pool_backend_1 | nova | enabled | down | 2014-11-27T05:42:35.000000 | None |

you can see, the ubuntu-server7@pool_backend_1 is in down state
then i issued :
stack@ubuntu-server7:~/devstack$ cinder service-disable --reason 'not config anymore' ubuntu-server7@pool_backend_1 cinder-volume

to disable this service

while i checked output of service-list:
stack@ubuntu-server7:~/devstack$ cinder service-list
+------------------+---------------------------------+------+----------+-------+----------------------------+--------------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+---------------------------------+------+----------+-------+----------------------------+--------------------+
| cinder-scheduler | ubuntu-server7 | nova | enabled | up | 2014-11-27T22:19:19.000000 | None |
| cinder-volume | ubuntu-server7@pool_backend_1 | nova | disabled | up | 2014-11-27T22:19:17.000000 | not config anymore |

the service was disabled, but the state was set to up, which is not reasonable

Thanks
Peter

Peter Wang (peter.wang)
summary: - cinder service-disable show not change current state of cinder serivces
+ cinder service-disable should not change current state of cinder
+ serivces
Revision history for this message
Huang Zhiteng (zhiteng-huang) wrote :

This is unfortunately a side-effect of 'cinder service-disable' touching the service record in DB, which would update the 'updated_at' timestamp of the record, result in the awkward situation you've seen. But that doesn't really hurt much, it's just weird.

Changed in cinder:
status: New → Confirmed
importance: Undecided → Wishlist
Mike Perez (thingee)
tags: added: low-hanging-fruit
removed: cinder
Hideki Saito (saitou)
Changed in cinder:
assignee: nobody → Hideki Saito (saitou)
Revision history for this message
Daniel Snider (daniel-snider) wrote :

A couple minutes after 'cinder service-disable' the state of the cinder service will go back to down on it's own, although this still isn't optimal.

Hideki Saito (saitou)
Changed in cinder:
assignee: Hideki Saito (saitou) → nobody
Revision history for this message
Rushi Agrawal (rushiagr) wrote :

There is no way to fix this, unless we have another field which keeps track of 'update_at' time for service status entry only, otherwise there is no way to distinguish if the 'updated_at' entry is changed by 'service-disable' command, or due to the periodic updates..

I don't think this falls under low-hanging-fruit tag..

Changed in cinder:
assignee: nobody → harsh mishra (harsh-mishra)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/154124

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/160104

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by harsh mishra (<email address hidden>) on branch: master
Review: https://review.openstack.org/154124
Reason: local repository error ,starting a new thread

Revision history for this message
harsh mishra (harsh-mishra) wrote :

Waiting for completion of verification and workflow of my fix at:
https://review.openstack.org/160104

Changed in cinder:
importance: Wishlist → Medium
Revision history for this message
harsh mishra (harsh-mishra) wrote :

need verified,need workflow for this fix to be merged,what are ways to get them?

Changed in cinder:
milestone: none → next
Revision history for this message
harsh mishra (harsh-mishra) wrote :

I have targeted the fix for kilo-rc1 release but the change is not yet committed to Cinder ,is there anything I am missing

Changed in cinder:
milestone: next → kilo-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/160104
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ee2a0c890f5f62b4da24a0bae4918f7a1d0ad2d2
Submitter: Jenkins
Branch: master

commit ee2a0c890f5f62b4da24a0bae4918f7a1d0ad2d2
Author: Harsh Mishra <email address hidden>
Date: Sat Feb 28 06:54:43 2015 +0000

    Fix for inconsistent cinder-services state change

    This bug was introduced to cinder as a side effect of fix :
    18365 (https://review.openstack.org/#/c/18365/).
    API “service_update” is used by periodic and manual service-enable/
    disable to update cinder-services table. Due to this ‘updated_at’
    column of cinder-services table receives an update for both periodic
    updates and manual service-enable/disable. Since state of any cinder
    service is calculated based on the ‘updated_at’ column of the service,
    we get inconsistent state transitions if a service is manually
    enabled/disabled and a service-list command is launched before
    “CONF.service_down_time”, for services which are already in a “down”
    state.
    This fix introduces a new column: “modified_at” in cinder-services
    table. As per the fix all the periodic updates will be updated in
    ‘updated_at’ column and manual service-enable/disable in
    “modified_at” column. This will result in correct state transitions
    as ‘updated_at’ column will only contains time value for periodic
    updates which in turn would be used to calculate the state of the
    cinder-service if it is ‘up’/’down’. Also, the time for any service
    receiving an update will be displayed as “updated_at” or
    “modified_at”, which ever is latest.

    Closes-bug: 1397167

    Change-Id: Iaee4a9cec035dd60c9de04699b1135ee306f6257

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in cinder:
milestone: kilo-rc1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.