| 2014-11-20 17:50:28 |
Denis M. |
description |
From server side there's no way to make device path as 'NoneType'. And if taskmanager.conf would contain something like:
[{datastore}]
device_path = /there/no/such/directory/at/all/volumes/are/disabled/
Guest will take it and will try to do next:
if device_path: # as you can see this check will work proprely
......
Then guest will do next:
...
device.unmount_device(device_path)
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L138-L145 it'll work because mount_points will an empty list
Then guest will do:
device.format()
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L108-L115
and so on.
See
https://github.com/openstack/trove/blob/master/trove/guestagent/datastore/mysql/manager.py#L124
Proper solution would be adding a check that tries to find out if 'device_path' really exist.
Here's a log
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Traceback (most recent call last):
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 62, in _check_device_exists
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume attempts=num_tries)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/processutils.py", line 200, in execute
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume cmd=sanitized_cmd)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume ProcessExecutionError: Unexpected error while running command.
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Command: sudo blockdev --getsize64 /dev/sdb
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Exit code: 1
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stdout: u''
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stderr: u'blockdev: cannot open /dev/sdb: No such file or directory\n'
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume
2014-11-20 17:23:04.560 ERROR trove.openstack.common.rpc.amqp [req-a9c3deca-0032-4c10-a493-b643ce3c7e11 cdf5140dde1a46408b3a6c88e9919449 45a970419c0e4095870f18f13b90c5f5] Exception during message handling
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp Traceback (most recent call last):
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp **args)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/manager.py", line 127, in prepare
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp device.format()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 104, in format
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self._check_device_exists()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 66, in _check_device_exists
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self.device_path)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp GuestError: InvalidDevicePath(path=/dev/sdb) |
From server side there's no way to make device path as 'NoneType'. And if taskmanager.conf would contain something like:
[{datastore}]
device_path = /there/no/such/directory/at/all/volumes/are/disabled/
Guest will take it and will try to do next:
if device_path: # as you can see this check will work proprely
......
Then guest will do next:
...
device.unmount_device(device_path)
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L138-L145 it'll work because mount_points will an empty list
Then guest will do:
device.format()
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L108-L115
and so on.
See
https://github.com/openstack/trove/blob/master/trove/guestagent/datastore/mysql/manager.py#L124
Proper solution would be adding a check that tries to find out if 'device_path' really exist.
Here's a log
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Traceback (most recent call last):
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 62, in _check_device_exists
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume attempts=num_tries)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/processutils.py", line 200, in execute
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume cmd=sanitized_cmd)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume ProcessExecutionError: Unexpected error while running command.
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Command: sudo blockdev --getsize64 /dev/sdb
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Exit code: 1
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stdout: u''
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stderr: u'blockdev: cannot open /dev/sdb: No such file or directory\n'
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume
2014-11-20 17:23:04.560 ERROR trove.openstack.common.rpc.amqp [req-a9c3deca-0032-4c10-a493-b643ce3c7e11 cdf5140dde1a46408b3a6c88e9919449 45a970419c0e4095870f18f13b90c5f5] Exception during message handling
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp Traceback (most recent call last):
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp **args)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/manager.py", line 127, in prepare
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp device.format()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 104, in format
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self._check_device_exists()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 66, in _check_device_exists
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self.device_path)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp GuestError: InvalidDevicePath(path=/dev/sdb)
more infor would be useful:
Here's a prepare message from Taskmanager
2014-11-20 17:22:52.481 DEBUG trove.openstack.common.rpc.amqp [-] received {u'_context_domain': None, u'_context_request_id': u'req-a9c3deca-0032-4c10-a493-b643ce3c7e11', u'args': {u'device_path': u'/dev/sdb', u'users': [], u'cluster_config': None, u'overrides': u'[mysqld]\n', u'memory_mb': 2048, u'mount_point': u'/var/lib/mysql', u'root_password': '<SANITIZED>', u'backup_info': None, u'databases': [], u'config_contents': u'[client]\nport = 3306\n\n[mysqld_safe]\nnice = 0\n\n[mysqld]\nuser = mysql\nport = 3306\nbasedir = /usr\ndatadir = /var/lib/mysql\n####tmpdir = /tmp\ntmpdir = /var/tmp\npid_file = /var/run/mysqld/mysqld.pid\nskip-external-locking = 1\nkey_buffer_size = 200M\nmax_allowed_packet = 4M\nthread_stack = 192K\nthread_cache_size = 16\nmyisam-recover = BACKUP\nquery_cache_type = 1\nquery_cache_limit = 1M\nquery_cache_size = 32M\ninnodb_data_file_path = ibdata1:10M:autoextend\ninnodb_buffer_pool_size = 600M\ninnodb_file_per_table = 1\ninnodb_log_files_in_group = 2\ninnodb_log_file_size=50M\ninnodb_log_buffer_size=25M\nconnect_timeout = 15\nwait_timeout = 120\njoin_buffer_size = 1M\nread_buffer_size = 512K\nread_rnd_buffer_size = 512K\nsort_buffer_size = 1M\ntmp_table_size = 64M\nmax_heap_table_size = 64M\ntable_open_cache = 1024\ntable_definition_cache = 1024\nopen_files_limit = 2048\nmax_user_connections = 400\nmax_connections = 400\ndefault_storage_engine = innodb\nlocal-infile = 0\nserver_id = 1481402592\n\n[mysqldump]\nquick = 1\nquote-names = 1\nmax_allowed_packet = 16M\n\n[isamchk]\nkey_buffer = 16M\n\n!includedir /etc/mysql/conf.d/', u'packages': [u'mysql-server-5.5']}, u'_context_read_only': False, u'_context_service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'id': u'010cdb89e9f3444991e63ac835bfea90'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'id': u'16f6f61521854995be8fd445d5e766c3'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.200:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.250:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://172.16.43.250:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'id': u'966a745b106e4773bb642ebf081077bb'}], u'type': u'database', u'name': u'trove'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'120ce1a8d2594ab9979f3b1ef55c577a'}], u'type': u'application_catalog', u'name': u'murano'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:9292', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:9292', u'internalURL': u'http://192.168.200.2:9292', u'id': u'202e005505d34c3ba98cf6982c1a508a'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'id': u'280601e810324bb7be9dec99a48fcb71'}], u'type': u'data_processing', u'name': u'sahara'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8000/v1/', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8000/v1/', u'internalURL': u'http://192.168.200.2:8000/v1/', u'id': u'04553282a4664efba700480308733b3e'}], u'type': u'cloudformation', u'name': u'heat-cfn'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8777', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8777', u'internalURL': u'http://192.168.200.2:8777', u'id': u'6b19a991ca714f05ac7a0f4d176bfca6'}], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'24c76d92308f4d9cbb90b810946e4f71'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8773/services/Admin', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8773/services/Cloud', u'internalURL': u'http://192.168.200.2:8773/services/Cloud', u'id': u'077944c11fd9428faf541b219f08b78a'}], u'type': u'ec2', u'name': u'nova_ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'3543724e96174a7297baa9a6f8f48946'}], u'type': u'orchestration', u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:35357/v2.0', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:5000/v2.0', u'internalURL': u'http://192.168.200.2:5000/v2.0', u'id': u'd4d0ac8d84c145279f54858172994570'}], u'type': u'identity', u'name': u'keystone'}], u'_context_user_identity': u'cdf5140dde1a46408b3a6c88e9919449 45a970419c0e4095870f18f13b90c5f5 - - -', u'_context_show_deleted': False, u'_context_tenant': u'45a970419c0e4095870f18f13b90c5f5', u'_context_auth_token': '<SANITIZED>', u'_context_project_domain': None, u'_context_is_admin': True, u'version': u'1.0', u'_context_marker': None, u'_context_instance_uuid': None, u'_context_user': u'cdf5140dde1a46408b3a6c88e9919449', u'_unique_id': u'248a9a5a26054a1e95f3b6bbbb3435aa', u'_context_user_domain': None, u'namespace': None, u'method': u'prepare', u'_context_limit': None} from (pid=1530) _safe_log /usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/common.py:280
As you can see even if volumes are disabled 'mount_point' and 'device_path' are not type of None. |
|
