Guest performs incorrect check of device path

Bug #1394682 reported by Denis M.
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
New
Undecided
Unassigned

Bug Description

From server side there's no way to make device path as 'NoneType'. And if taskmanager.conf would contain something like:

[{datastore}]
device_path = /there/no/such/directory/at/all/volumes/are/disabled/

Guest will take it and will try to do next:

if device_path: # as you can see this check will work proprely
......

Then guest will do next:
...
device.unmount_device(device_path)
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L138-L145 it'll work because mount_points will an empty list

Then guest will do:
device.format()
https://github.com/openstack/trove/blob/master/trove/guestagent/volume.py#L108-L115

and so on.

See
https://github.com/openstack/trove/blob/master/trove/guestagent/datastore/mysql/manager.py#L124

Proper solution would be adding a check that tries to find out if 'device_path' really exist.

Here's a log

2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Traceback (most recent call last):
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 62, in _check_device_exists
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume attempts=num_tries)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/processutils.py", line 200, in execute
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume cmd=sanitized_cmd)
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume ProcessExecutionError: Unexpected error while running command.
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Command: sudo blockdev --getsize64 /dev/sdb
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Exit code: 1
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stdout: u''
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume Stderr: u'blockdev: cannot open /dev/sdb: No such file or directory\n'
2014-11-20 17:23:04.549 TRACE trove.guestagent.volume
2014-11-20 17:23:04.560 ERROR trove.openstack.common.rpc.amqp [req-a9c3deca-0032-4c10-a493-b643ce3c7e11 cdf5140dde1a46408b3a6c88e9919449 45a970419c0e4095870f18f13b90c5f5] Exception during message handling
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp Traceback (most recent call last):
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp **args)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/datastore/mysql/manager.py", line 127, in prepare
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp device.format()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 104, in format
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self._check_device_exists()
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/trove/guestagent/volume.py", line 66, in _check_device_exists
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp self.device_path)
2014-11-20 17:23:04.560 TRACE trove.openstack.common.rpc.amqp GuestError: InvalidDevicePath(path=/dev/sdb)

more infor would be useful:
Here's a prepare message from Taskmanager

2014-11-20 17:22:52.481 DEBUG trove.openstack.common.rpc.amqp [-] received {u'_context_domain': None, u'_context_request_id': u'req-a9c3deca-0032-4c10-a493-b643ce3c7e11', u'args': {u'device_path': u'/dev/sdb', u'users': [], u'cluster_config': None, u'overrides': u'[mysqld]\n', u'memory_mb': 2048, u'mount_point': u'/var/lib/mysql', u'root_password': '<SANITIZED>', u'backup_info': None, u'databases': [], u'config_contents': u'[client]\nport = 3306\n\n[mysqld_safe]\nnice = 0\n\n[mysqld]\nuser = mysql\nport = 3306\nbasedir = /usr\ndatadir = /var/lib/mysql\n####tmpdir = /tmp\ntmpdir = /var/tmp\npid_file = /var/run/mysqld/mysqld.pid\nskip-external-locking = 1\nkey_buffer_size = 200M\nmax_allowed_packet = 4M\nthread_stack = 192K\nthread_cache_size = 16\nmyisam-recover = BACKUP\nquery_cache_type = 1\nquery_cache_limit = 1M\nquery_cache_size = 32M\ninnodb_data_file_path = ibdata1:10M:autoextend\ninnodb_buffer_pool_size = 600M\ninnodb_file_per_table = 1\ninnodb_log_files_in_group = 2\ninnodb_log_file_size=50M\ninnodb_log_buffer_size=25M\nconnect_timeout = 15\nwait_timeout = 120\njoin_buffer_size = 1M\nread_buffer_size = 512K\nread_rnd_buffer_size = 512K\nsort_buffer_size = 1M\ntmp_table_size = 64M\nmax_heap_table_size = 64M\ntable_open_cache = 1024\ntable_definition_cache = 1024\nopen_files_limit = 2048\nmax_user_connections = 400\nmax_connections = 400\ndefault_storage_engine = innodb\nlocal-infile = 0\nserver_id = 1481402592\n\n[mysqldump]\nquick = 1\nquote-names = 1\nmax_allowed_packet = 16M\n\n[isamchk]\nkey_buffer = 16M\n\n!includedir /etc/mysql/conf.d/', u'packages': [u'mysql-server-5.5']}, u'_context_read_only': False, u'_context_service_catalog': [{u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8774/v2/45a970419c0e4095870f18f13b90c5f5', u'id': u'010cdb89e9f3444991e63ac835bfea90'}], u'type': u'compute', u'name': u'nova'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8776/v2/45a970419c0e4095870f18f13b90c5f5', u'id': u'16f6f61521854995be8fd445d5e766c3'}], u'type': u'volumev2', u'name': u'cinderv2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.200:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.250:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://172.16.43.250:8779/v1.0/45a970419c0e4095870f18f13b90c5f5', u'id': u'966a745b106e4773bb642ebf081077bb'}], u'type': u'database', u'name': u'trove'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8082/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'120ce1a8d2594ab9979f3b1ef55c577a'}], u'type': u'application_catalog', u'name': u'murano'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:9292', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:9292', u'internalURL': u'http://192.168.200.2:9292', u'id': u'202e005505d34c3ba98cf6982c1a508a'}], u'type': u'image', u'name': u'glance'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8386/v1.1/45a970419c0e4095870f18f13b90c5f5', u'id': u'280601e810324bb7be9dec99a48fcb71'}], u'type': u'data_processing', u'name': u'sahara'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8000/v1/', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8000/v1/', u'internalURL': u'http://192.168.200.2:8000/v1/', u'id': u'04553282a4664efba700480308733b3e'}], u'type': u'cloudformation', u'name': u'heat-cfn'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8777', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8777', u'internalURL': u'http://192.168.200.2:8777', u'id': u'6b19a991ca714f05ac7a0f4d176bfca6'}], u'type': u'metering', u'name': u'ceilometer'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8776/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'24c76d92308f4d9cbb90b810946e4f71'}], u'type': u'volume', u'name': u'cinder'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8773/services/Admin', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8773/services/Cloud', u'internalURL': u'http://192.168.200.2:8773/services/Cloud', u'id': u'077944c11fd9428faf541b219f08b78a'}], u'type': u'ec2', u'name': u'nova_ec2'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'internalURL': u'http://192.168.200.2:8004/v1/45a970419c0e4095870f18f13b90c5f5', u'id': u'3543724e96174a7297baa9a6f8f48946'}], u'type': u'orchestration', u'name': u'heat'}, {u'endpoints_links': [], u'endpoints': [{u'adminURL': u'http://192.168.200.2:35357/v2.0', u'region': u'RegionOne', u'publicURL': u'http://172.16.43.42:5000/v2.0', u'internalURL': u'http://192.168.200.2:5000/v2.0', u'id': u'd4d0ac8d84c145279f54858172994570'}], u'type': u'identity', u'name': u'keystone'}], u'_context_user_identity': u'cdf5140dde1a46408b3a6c88e9919449 45a970419c0e4095870f18f13b90c5f5 - - -', u'_context_show_deleted': False, u'_context_tenant': u'45a970419c0e4095870f18f13b90c5f5', u'_context_auth_token': '<SANITIZED>', u'_context_project_domain': None, u'_context_is_admin': True, u'version': u'1.0', u'_context_marker': None, u'_context_instance_uuid': None, u'_context_user': u'cdf5140dde1a46408b3a6c88e9919449', u'_unique_id': u'248a9a5a26054a1e95f3b6bbbb3435aa', u'_context_user_domain': None, u'namespace': None, u'method': u'prepare', u'_context_limit': None} from (pid=1530) _safe_log /usr/local/lib/python2.7/dist-packages/trove/openstack/common/rpc/common.py:280

As you can see even if volumes are disabled 'mount_point' and 'device_path' are not type of None.

Denis M. (dmakogon)
description: updated
Revision history for this message
Tim Simpson (tim-simpson) wrote :

I agree, device_path should be none. Or even better, the guest should have an option to enable or disable volume support.

However it seems like if the device_path is an empty string it won't hit the volume code path and cause any of these problems, so I feel like I'm missing what the root issue you're addressing is.

Denis M. (dmakogon)
description: updated
Changed in trove:
assignee: nobody → Victoria Martínez de la Cruz (vkmc)
Revision history for this message
Sushil Kumar (sushil-kumar2) wrote :

What do you mean by
device_path = /there/no/such/directory/at/all/volumes/are/disabled/

Either you have volume_support=True, where one uses cinder volumes
or
it is volume_support=False, where one uses ephemeral storage.

And in both cases we need to have a device path to be used, so I do not see the point of getting None for device_path.

The probable reason of failure of log, what I feel is the changed value of device_path, which is visible as /dev/sdb, which by default for trove code is /dev/vdb.

So can you please elaborate more on None usage of device_path

Revision history for this message
Amrith Kumar (amrith) wrote :

vkmc, do you have an update on a fix for this?

Revision history for this message
Amrith Kumar (amrith) wrote :

vkmc, any updates on this bug?

Changed in trove:
assignee: Victoria Martinez de la Cruz (vkmc) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.