TypeError: expected a string or other character buffer object with autotune=true

[Nobuto Murata]

$ juju config ceph-osd autotune=true

Then,
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.juju-log Looks like /dev/sdb is in use, skipping.
unit-ceph-osd-0: 04:11:07 DEBUG unit.ceph-osd/0.juju-log block device /dev/sdb uuid is None. Unable to save to hdparm.conf
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.juju-log Tuning device /dev/sdb
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.juju-log Setting read ahead to 256 for device /dev/sdb
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.juju-log max_sectors_kb match max_hw_sectors_kb. No change needed for device: /dev/sdb
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed Traceback (most recent call last):
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 517, in <module>
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed hooks.execute(sys.argv)
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/core/hookenv.py", line 715, in execute
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed self._hooks[hook_name]()
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed return f(*args, **kwargs)
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 332, in config_changed
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed prepare_disks_and_activate()
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 357, in prepare_disks_and_activate
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed ceph.tune_dev(dev)
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "lib/ceph/__init__.py", line 385, in tune_dev
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed persist_settings(settings_dict=save_settings_dict)
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed File "lib/ceph/__init__.py", line 204, in persist_settings
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed hdparm.write(render_template('hdparm.conf', settings_dict))
unit-ceph-osd-0: 04:11:07 INFO unit.ceph-osd/0.config-changed TypeError: expected a string or other character buffer object
unit-ceph-osd-0: 04:11:07 ERROR juju.worker.uniter.operation hook "config-changed" failed: exit status 1

[Nobuto Murata (nobuto)]

Please note that those OSD devices are behind a RAID card.

[Nobuto Murata (nobuto)]

NOTE: will attach `blkid -o export` on the next deployment.

[Nobuto Murata (nobuto)]

output of `blkid -o export`

[Nobuto Murata (nobuto)]

https://review.openstack.org/#/q/topic:bug/1660240

[Billy Olsen (billy-olsen)]

Recreated this tonight. The problem is that the write call is being passed a charmhelpers.core.services.helpers.TemplateCallback object instead of a string object.

[Billy Olsen (billy-olsen)]

As I review this code around the tune_dev a bit further, various questions come to mind. For one, the UUID of the /dev/<disk> generally isn't set as that's set on the partition not on the device itself, so I don't expect the disk itself to have a UUID. Commit https://github.com/openstack/charms.ceph/commit/69b691edafae7e10be878c41c34ab2fbfecaedfc should actually prevent this exception from being raised as the UUID will be None and therefore will bail on the autotune, but it makes the autotune ineffective as I believe that in general the UUIDs for the disk will be None. However, this code change was not synced into the ceph-osd charm recently and therefore the exception is still encountered in the latest charm (17.02 release, ceph-osd-241).

The documentation around hdparm seems to imply using the /dev/disk/by-id/<disk-id> for settings in the hdparm.conf file, but I'm not sure how to determine this value other than checking the various links in /dev/disk/by-id and resolving the os.path.realpath to the device in question. So instead of searching for the UUID, the code should probably be searching for the disk's id instead. Also, there are some classes of disks that should be skipped because the hdparm utility doesn't actually support them; namely these include certain virtual disks which are not attached via SCSI, SAS, or SATA. This limitation should really only affect testing scenarios though.

Finally, the tuning in question sets the read_ahead value for the disk to 256. However, all systems I've seen in practice set this value to 256. I'm curious if this might be the default value but admit to not knowing yet exactly how the default read_ahead is chosen.

[Billy Olsen (billy-olsen)]

My mistake on the setting of the read_ahead. I missed the bit of code in which it sets the read_ahead by default, then it updates the read_ahead based on the disk's geometry.

[Billy Olsen (billy-olsen)]

Correction, this bug is not necessarily fixed for raid devices which may have a UUID on the disk itself. It will get the UUID but will still fail because the render_template code is incorrect for generating the hwparms.conf file. I'll propose a fix for that as well.

[Frode Nordahl (fnordahl)]

2017-06-15 09:27:55 INFO config-changed Traceback (most recent call last):
2017-06-15 09:27:55 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 524, in <module>
2017-06-15 09:27:55 INFO config-changed hooks.execute(sys.argv)
2017-06-15 09:27:55 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-06-15 09:27:55 INFO config-changed self._hooks[hook_name]()
2017-06-15 09:27:55 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-06-15 09:27:55 INFO config-changed return f(*args, **kwargs)
2017-06-15 09:27:55 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 335, in config_changed
2017-06-15 09:27:55 INFO config-changed prepare_disks_and_activate()
2017-06-15 09:27:55 INFO config-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/config-changed", line 361, in prepare_disks_and_activate
2017-06-15 09:27:55 INFO config-changed ceph.tune_dev(dev)
2017-06-15 09:27:55 INFO config-changed File "lib/ceph/__init__.py", line 405, in tune_dev
2017-06-15 09:27:55 INFO config-changed persist_settings(settings_dict=save_settings_dict)
2017-06-15 09:27:55 INFO config-changed File "lib/ceph/__init__.py", line 224, in persist_settings
2017-06-15 09:27:55 INFO config-changed hdparm.write(render_template('hdparm.conf', settings_dict))
2017-06-15 09:27:55 INFO config-changed TypeError: expected a string or other character buffer object
2017-06-15 09:27:55 ERROR juju.worker.uniter.operation runhook.go:107 hook "config-changed" failed: exit status 1

[Billy Olsen (billy-olsen)]

This is fixed in master in commit 0e0e3933e363d10e6d7569682c14a881bd8ae238 on charms.ceph library (https://review.openstack.org/#/c/444017/). Unfortunately, there's no stable branch created with the charms.ceph in order to cherry-pick it to and then sync it in properly.