| 2013-06-19 02:48:01 |
kaitian521 |
bug |
|
|
added bug |
| 2013-06-19 06:51:00 |
Huang Zhiteng |
cinder: status |
New |
Invalid |
|
| 2013-06-19 06:51:35 |
Huang Zhiteng |
cinder: status |
Invalid |
Incomplete |
|
| 2013-06-19 07:28:46 |
kaitian521 |
description |
when I have a bad cinder-volume service, cinder will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
the last line ,should be
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
cinder will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
the last line ,should be
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
|
| 2013-06-19 07:31:11 |
kaitian521 |
tags |
|
cinder-scheduler |
|
| 2013-06-19 07:31:13 |
kaitian521 |
description |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
cinder will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
the last line ,should be
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
|
| 2013-06-19 07:31:24 |
kaitian521 |
cinder: status |
Incomplete |
New |
|
| 2013-06-19 07:52:53 |
Huang Zhiteng |
cinder: status |
New |
Incomplete |
|
| 2013-06-19 08:25:00 |
kaitian521 |
description |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
#######################################################################################
SO it is very easy to reproduce it:
step 1: change our hostname to others "hostname other"
step 2 : restart all cinder service : "cinder-api, cinder-manage, cinder-volume"
Then you can get some bad cinder-volume service as above.
########################################################################################
When running "cinder create 1"
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
|
| 2013-06-19 08:27:28 |
kaitian521 |
cinder: status |
Incomplete |
New |
|
| 2013-06-19 08:28:43 |
kaitian521 |
description |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
#######################################################################################
SO it is very easy to reproduce it:
step 1: change our hostname to others "hostname other"
step 2 : restart all cinder service : "cinder-api, cinder-manage, cinder-volume"
Then you can get some bad cinder-volume service as above.
########################################################################################
When running "cinder create 1"
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
#######################################################################################
SO it is very easy to reproduce it:
step 1: change our hostname to others "hostname other"
step 2 : restart all cinder service : "cinder-api, cinder-scheduler, cinder-volume"
step 3 : cinder-manage service list
Then you can get some bad cinder-volume service as above.
########################################################################################
When running "cinder create 1"
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
|
| 2013-06-24 02:30:46 |
renminmin |
cinder: status |
New |
Incomplete |
|
| 2013-06-24 02:33:13 |
renminmin |
cinder: status |
Incomplete |
New |
|
| 2013-07-03 08:38:51 |
Vincent Hou |
cinder: status |
New |
Incomplete |
|
| 2013-07-03 08:38:57 |
Vincent Hou |
cinder: status |
Incomplete |
Confirmed |
|
| 2013-07-04 15:31:10 |
Haomai Wang |
cinder: assignee |
|
Haomai Wang (haomai) |
|
| 2013-07-04 15:56:00 |
OpenStack Infra |
cinder: status |
Confirmed |
In Progress |
|
| 2013-07-04 20:44:02 |
Huang Zhiteng |
summary |
cinder/scheduler/host_manager.py , line 268, in get_all_host_states |
Cinder host manager doesn't remove disabled/stopped service from host_state_map |
|
| 2013-07-04 20:46:10 |
Huang Zhiteng |
description |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
#######################################################################################
SO it is very easy to reproduce it:
step 1: change our hostname to others "hostname other"
step 2 : restart all cinder service : "cinder-api, cinder-scheduler, cinder-volume"
step 3 : cinder-manage service list
Then you can get some bad cinder-volume service as above.
########################################################################################
When running "cinder create 1"
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
##################################################################################
When cinder-scheduler decide which cinder-volume is bad, it will log a warning log:
LOG.warn(_("volume service is down or disabled. "
#"(host: %s)") % host)
BUT, this is wrong; the right should be this:
for service in volume_services:
host = service['host']
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("volume service is down or disabled. "
"(host: %s)" % host) )
#"(host: %s)") % host)
continue |
when I have a bad cinder-volume service,
cinder-manage service list
cinder-volume localhost nova enabled :-) 2013-06-19 07:20:17
cinder-scheduler localhost nova enabled :-) 2013-06-19 07:20:17
cinder-volume other nova enabled XXX 2013-06-19 07:20:17
cinder-scheduler other nova enabled XXX 2013-06-19 07:20:17
#######################################################################################
SO it is very easy to reproduce it:
step 1: change our hostname to others "hostname other"
step 2 : restart all cinder service : "cinder-api, cinder-scheduler, cinder-volume"
step 3 : cinder-manage service list
Then you can get some bad cinder-volume service as above.
########################################################################################
When running "cinder create 1"
cinder scheduler will log an warning:
Traceback (most recent call last):
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp **args)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 115, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp context, ex, request_spec)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/manager.py", line 104, in create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 64, in schedule_create_volume
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/filter_scheduler.py", line 197, in _schedule
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py", line 268, in get_all_host_states
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp "(host: %s)") % host)
2013-06-18 04:01:38.771 54708 TRACE cinder.openstack.common.rpc.amqp TypeError: not all arguments converted during string formatting
Possible solution is to remove disabled/stopped service from host_state_map() or simple clear the whole map.
########################################
for service in volume_services:
if not utils.service_is_up(service) or service['disabled']:
LOG.warn(_("service is down or disabled. Host name :%s")
% service['host'])
_host = self.host_state_map.get(service['host'])
if _host:
self.host_state_map.pop(service['host'])
continue
host = service['host']
########################################
The reason is that get_all_host_states method is use to get all service state and cinder scheduler choose the cinder-volume service from them.
Then when cinder-volume service is down or disabled, the state of bad one need remove from host_state_map list. |
|
| 2013-07-09 04:07:43 |
Haomai Wang |
cinder: status |
In Progress |
Fix Committed |
|
| 2013-07-17 11:08:47 |
Thierry Carrez |
cinder: status |
Fix Committed |
Fix Released |
|
| 2013-07-17 11:08:47 |
Thierry Carrez |
cinder: milestone |
|
havana-2 |
|
| 2013-10-17 10:23:43 |
Thierry Carrez |
cinder: milestone |
havana-2 |
2013.2 |
|
| 2014-10-17 10:31:49 |
Lukas Pustina |
bug |
|
|
added subscriber Lukas Pustina |
