Cinder

cinder.conf changes doesn't clean up service list

Bug #1340764 reported by Rajini Karthik
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Won't Fix
Wishlist
Unassigned

Bug Description

In a muti-backend cinder configuration , volume backend section name changes to cinder.conf file is not hanndled properly and causes volume creation errors.

For e.g: If the section name changes and c-vol service is restarted, the 'State' of the service is down , but the 'Status' is enabled. You have to manually disable the service using service-disable command for it to be invalid(or delete from database using brute force).

Before:
enabled_backends=eqlx1
[eqlx1]
volume_driver=<drivername>
...
volume_backend_name=eqlx1

After:
enabled_backends=eqlx2
[eqlx2]
volume_driver=<drivername>
...
volume_backend_name=eqlx2

stack@au-ubuntu-devstall-all-2:~/devstack$ cinder service-list
+------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
| cinder-scheduler | au-ubuntu-devstall-all-2 | nova | enabled | up | 2014-07-11T13:45:42.000000 | None |
| cinder-volume | au-ubuntu-devstall-all-2 | nova | disabled | down | 2014-07-10T14:20:09.000000 | usingmultibackend |
| cinder-volume | au-ubuntu-devstall-all-2@eqlx1 | nova | enabled | down | 2014-07-11T13:43:32.000000 | None |
| cinder-volume | au-ubuntu-devstall-all-2@eqlx2 | nova | enabled | up | 2014-07-11T13:45:39.000000 | None |
+------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+

See original description
Rajini Karthik (rajini-karthik)
description: updated
description: updated
Rajini Karthik (rajini-karthik)
description: updated
Revision history for this message
John Griffith (john-griffith) wrote :

The name you're setting in the config file is used to identify the service host (or our version of a fake-service host at any rate). This is expected behavior that you can't go in and *change* the host name on existing volumes.

In order to do this sort of thing you need to use the cinder-manage update_host command
(Change: 2f7c40421627b267512b88ecb0cbaf6aedc06ea7)

Revision history for this message
John Griffith (john-griffith) wrote :

Kinda need something like a "service remove" I think.

Revision history for this message
harsh mishra (harsh-mishra) wrote :

I think we should follow white listing in this wherein cinder-scheduler should compare the latest hosts in cinder.conf when restarted with the ones in the cinder-services db .If a service-host-name is not found in cinder.conf we should remove the entry from the database and related entries.

This being done only when cinder-scheduler service is restarted,will enable us to view the current services active and also give a clean and consistent db.

harsh mishra (harsh-mishra)
Changed in cinder:
importance: Undecided → Wishlist
assignee: nobody → harsh mishra (harsh-mishra)
status: New → In Progress
Revision history for this message
Rajini Karthik (rajini-karthik) wrote :

Harsh ,
Did you see john's comments abot service remove?

What are the related db entries? What happens to the existing volumes created under the old hostname?

Revision history for this message
John Griffith (john-griffith) wrote :

@harsh
So keep in mind the other use cases here (not multi-backend, but multiple cinder volume nodes). I still say that the best solution to what you're discussing here seems to be a "service remove".

Migrate is another issue altogether, but we already mentioned ways to update host column to make that work. If changing "names" of services was something that was a regular occurrence I'd probably have more of a sense of urgency here, but it's not something I think should be done on a regular basis (if ever).

Revision history for this message
arkady kanevsky (arkady-kanevsky) wrote : RE: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

I think you just sent it to me...

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ram, Rajini
Sent: Monday, February 02, 2015 12:54 PM
To: Kanevsky, Arkady
Subject: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

Harsh ,
Did you see john's comments abot service remove?

What are the related db entries? What happens to the existing volumes created under the old hostname?

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1340764

Title:
  cinder.conf changes doesn't clean up service list

Status in Cinder:
  In Progress

Bug description:
  In a muti-backend cinder configuration , volume backend section name
  changes to cinder.conf file is not hanndled properly and causes volume
  creation errors.

  For e.g: If the section name changes and c-vol service is restarted,
  the 'State' of the service is down , but the 'Status' is enabled. You
  have to manually disable the service using service-disable command for
  it to be invalid(or delete from database using brute force).

  Before:
  enabled_backends=eqlx1
  [eqlx1]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx1

  After:
  enabled_backends=eqlx2
  [eqlx2]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx2

  stack@au-ubuntu-devstall-all-2:~/devstack$ cinder service-list
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | cinder-scheduler | au-ubuntu-devstall-all-2 | nova | enabled | up | 2014-07-11T13:45:42.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2 | nova | disabled | down | 2014-07-10T14:20:09.000000 | usingmultibackend |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx1 | nova | enabled | down | 2014-07-11T13:43:32.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx2 | nova | enabled | up | 2014-07-11T13:45:39.000000 | None |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1340764/+subscriptions

Revision history for this message
Rajini Karthik (rajini-karthik) wrote :
Download full text (5.0 KiB)

I added comments to the bug, it sends a copy to everyone that subscribed to it. I'll forward to harshas well

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Kanevsky, Arkady
Sent: Monday, February 02, 2015 1:25 PM
To: Ram, Rajini
Subject: RE: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

I think you just sent it to me...

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ram, Rajini
Sent: Monday, February 02, 2015 12:54 PM
To: Kanevsky, Arkady
Subject: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

Harsh ,
Did you see john's comments abot service remove?

What are the related db entries? What happens to the existing volumes created under the old hostname?

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1340764

Title:
  cinder.conf changes doesn't clean up service list

Status in Cinder:
  In Progress

Bug description:
  In a muti-backend cinder configuration , volume backend section name
  changes to cinder.conf file is not hanndled properly and causes volume
  creation errors.

  For e.g: If the section name changes and c-vol service is restarted,
  the 'State' of the service is down , but the 'Status' is enabled. You
  have to manually disable the service using service-disable command for
  it to be invalid(or delete from database using brute force).

  Before:
  enabled_backends=eqlx1
  [eqlx1]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx1

  After:
  enabled_backends=eqlx2
  [eqlx2]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx2

  stack@au-ubuntu-devstall-all-2:~/devstack$ cinder service-list
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | cinder-scheduler | au-ubuntu-devstall-all-2 | nova | enabled | up | 2014-07-11T13:45:42.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2 | nova | disabled | down | 2014-07-10T14:20:09.000000 | usingmultibackend |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx1 | nova | enabled | down | 2014-07-11T13:43:32.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx2 | nova | enabled | up | 2014-07-11T13:45:39.000000 | None |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1340764/+subscriptions

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1340764

Title:
  cinder.conf changes doesn't clean up service list

Status in ...

Read more...

Revision history for this message
arkady kanevsky (arkady-kanevsky) wrote :
Download full text (7.4 KiB)

+1

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ram, Rajini
Sent: Monday, February 02, 2015 1:36 PM
To: Kanevsky, Arkady
Subject: RE: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

I added comments to the bug, it sends a copy to everyone that subscribed to it. I'll forward to harshas well

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Kanevsky, Arkady
Sent: Monday, February 02, 2015 1:25 PM
To: Ram, Rajini
Subject: RE: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

I think you just sent it to me...

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Ram, Rajini
Sent: Monday, February 02, 2015 12:54 PM
To: Kanevsky, Arkady
Subject: [Bug 1340764] Re: cinder.conf changes doesn't clean up service list

Harsh ,
Did you see john's comments abot service remove?

What are the related db entries? What happens to the existing volumes created under the old hostname?

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1340764

Title:
  cinder.conf changes doesn't clean up service list

Status in Cinder:
  In Progress

Bug description:
  In a muti-backend cinder configuration , volume backend section name
  changes to cinder.conf file is not hanndled properly and causes volume
  creation errors.

  For e.g: If the section name changes and c-vol service is restarted,
  the 'State' of the service is down , but the 'Status' is enabled. You
  have to manually disable the service using service-disable command for
  it to be invalid(or delete from database using brute force).

  Before:
  enabled_backends=eqlx1
  [eqlx1]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx1

  After:
  enabled_backends=eqlx2
  [eqlx2]
  volume_driver=<drivername>
  ...
  volume_backend_name=eqlx2

  stack@au-ubuntu-devstall-all-2:~/devstack$ cinder service-list
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+
  | cinder-scheduler | au-ubuntu-devstall-all-2 | nova | enabled | up | 2014-07-11T13:45:42.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2 | nova | disabled | down | 2014-07-10T14:20:09.000000 | usingmultibackend |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx1 | nova | enabled | down | 2014-07-11T13:43:32.000000 | None |
  | cinder-volume | au-ubuntu-devstall-all-2@eqlx2 | nova | enabled | up | 2014-07-11T13:45:39.000000 | None |
  +------------------+--------------------------------+------+----------+-------+----------------------------+-------------------+

To manage notifications about this bug go to:
h...

Read more...

Revision history for this message
harsh mishra (harsh-mishra) wrote :

Hi Rajini and John :

Considering the use cases you have mentioned ,the service entries are really important in db and we should not delete it at startup, rather we should provide an API 'service-remove' as John suggested to remove the particular service on user-request.
I don't think we should completely leave it to the user's consent to delete a service without proper sanity checks.
Rather we should discuss similar use-cases(like if a volume/snapshot with the given host name is present) where the user should not be allowed to delete the service entry. Based on those use-cases we should filter request and proceed with delete.

Revision history for this message
harsh mishra (harsh-mishra) wrote :

Alternative approach
===================
We can also try to skip displaying backend cinder-services if it is 'down' and not present in cinder.conf file.
The backend service will be displayed to end-user only if the service comes to 'up' state .
 (check api skip_redundant_hosts)

 for svc in services:
            delta = now - (svc['updated_at'] or svc['created_at'])
            alive = abs(utils.total_seconds(delta)) <= CONF.service_down_time
            art = (alive and "up") or "down"
            active = 'enabled'
            if svc['disabled']:
                active = 'disabled'
        if art=="down":
          if self._skip_redundant_hosts(context,svc):
            continue
            ret_fields = {'binary': svc['binary'], 'host': svc['host'],
                          'zone': svc['availability_zone'],
                          'status': active, 'state': art,
                          'updated_at': svc['updated_at']}
            if detailed:
                ret_fields['disabled_reason'] = svc['disabled_reason']
            svcs.append(ret_fields)
        return {'services': svcs}

    def _skip_redundant_hosts(self,context,svc):
    part = svc['host'].partition('@')
    if (part[1] == '@') and (part[2] != '') and CONF.enabled_backends:
       backends=[]
           for backend in CONF.enabled_backends:
           backends.append(backend)
           if part[2] not in backends:
              return True
    return False

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Automatically unassigning due to inactivity.

Changed in cinder:
assignee: harsh mishra (harsh-mishra) → nobody
status: In Progress → Triaged
Revision history for this message
Ivan Kolodyazhny (e0ne) wrote :

We can't do it automatically. Please, use `cinder-manage service remove` command for it

Changed in cinder:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.