resize-volume fails after create from backup

[Doug Shelley]

I did these steps:

$ trove create mydb 100 --size 2 --datastore mysql --datastore-version mysql-5.5 --databases custdemo --users doug:password

I loaded a small schema and small amount of data into the custdemo database (I don't really believe this was material to the isuse)

$ trove backup-create <uuid> mybkup

I decided to delete the instance (so I had room on my compute :) )
$ trove delete <uuid of mydb>

Create a new instance from backup
$ trove create mynew 100 --size 2 --datastore mysql --backup <uuid of mybkup>

Resize the volume
$ trove resize-volume <uuid of mynew> 3

Instance never returns from "RESIZE" state

Noticed this in trove-guestagent.log

2014-08-29 16:47:39.652 945 DEBUG trove.openstack.common.processutils [req-44872
009-db57-498b-a141-58a664bacbfa 1afaeff0e5bb45cfa7616058b139b5fb c6d890d77f4b4ae
89ec044b1a5b5faf6 - - -] Running cmd (subprocess): sudo e2fsck -f -p /dev/vdb ex
ecute /usr/lib/python2.7/dist-packages/trove/openstack/common/processutils.py:15
8
2014-08-29 16:47:40.372 945 DEBUG trove.openstack.common.processutils [req-44872
009-db57-498b-a141-58a664bacbfa 1afaeff0e5bb45cfa7616058b139b5fb c6d890d77f4b4ae
89ec044b1a5b5faf6 - - -] Result was 1 execute /usr/lib/python2.7/dist-packages/t
rove/openstack/common/processutils.py:192
2014-08-29 16:47:40.374 945 ERROR trove.guestagent.volume [req-44872009-db57-498
b-a141-58a664bacbfa 1afaeff0e5bb45cfa7616058b139b5fb c6d890d77f4b4ae89ec044b1a5b
5faf6 - - -] Error resizing file system.
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume Traceback (most recent
 call last):
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume File "/usr/lib/pytho
n2.7/dist-packages/trove/guestagent/volume.py", line 122, in resize_fs
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume run_as_root=True,
root_helper="sudo")
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume File "/usr/lib/pytho
n2.7/dist-packages/trove/openstack/common/processutils.py", line 198, in execute
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume cmd=' '.join(cmd))
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume ProcessExecutionError:
 Unexpected error while running command.
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume Command: sudo e2fsck -
f -p /dev/vdb
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume Exit code: 1
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume Stdout: '/dev/vdb: /lo
st+found not found. CREATED.\n/dev/vdb: 129/131072 files (10.9% non-contiguous)
, 56675/524288 blocks\n'
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume Stderr: ''
2014-08-29 16:47:40.374 945 TRACE trove.guestagent.volume
2014-08-29 16:47:40.429 945 ERROR trove.openstack.common.rpc.amqp [req-44872009-
db57-498b-a141-58a664bacbfa 1afaeff0e5bb45cfa7616058b139b5fb c6d890d77f4b4ae89ec
044b1a5b5faf6 - - -] Exception during message handling

It looks like e2fsck is succeeding but return exit code 1 because of the "lost+found not found" warning. From what i can tell, Mysql doesn't like that there is a "lost+found" directory with its data and renames it.

[Simon Chang (changsimon)]

The db volume is mounted under /var/lib/mysql. The xtrabackup restore function (trove/guestagent/strategies/restore/mysql_impl.py) restores the lost+found directory in the volume as lost@002bfound. When resize volume runs, the e2fsck step raise a warning saying the lost+found directory is not found, because the earlier restore named the "lost+found" directory as "lost@002bfound".

[Simon Chang (changsimon)]

Related bug https://bugs.launchpad.net/trove/+bug/1370646

[Doug Shelley (0-doug)]

Given that the solution to this is going to be related to https://bugs.launchpad.net/trove/+bug/1370646 and it has been triaged to Kilo - I've moved this to Kilo.

[Nikhil Manchanda (slicknik)]

Hi Simon -- any updates on this?

[Simon Chang (changsimon)]

Hi Nikil, I've implemented the /var/lib/mysql/data sub-directory per recommendation (https://bugs.launchpad.net/trove/+bug/1370646). Need to finalize the patch and deal with this first https://bugs.launchpad.net/trove/+bug/1391639. Without addressing 1391639 first, resize-volume will fail anyway. That's why I haven't uploaded any patch here yet.

[Nikhil Manchanda (slicknik)]

Related to https://bugs.launchpad.net/trove/+bug/1370646

[Nikhil Manchanda (slicknik)]

Hi Simon -- any updates on this now that the fix for 1391639 has merged?
Thanks!

[Simon Chang (changsimon)]

Hi Nikhil, 1391639 would have addressed this issue. I'm closing this bug.