trove create of postgresql instance doesn't work, when used in conjunction with --backup option

[Nirav Shah]

Here are the steps to reproduce:
1. Create a postgresql instance:

trove create postgresql_instance 100 --datastore postgresql --datastore_version 9.3 --size 1 --databases db1 --users user1:password1

2. Verify it is ACTIVE:

trove list

3. Create a backup of this new instance:

trove backup-create <id> postgresql_instance-backup --description postgresql_instance-description

4. Verify backup is complete:

trove backup-list

5. Create a new instance, using this backup instance:

trove create postgresql_instance-restored 100 --datastore postgresql --datastore_version 9.3 --size 1 --backup <backup_id>

You will see the following error in the trove-guestagent log:

2015-01-13 19:23:20.293 1331 ERROR trove.openstack.common.rpc.amqp [req-ba4801b3-a894-4928-a795-d038967a9f6d 1297a95379fa4a529dc5cefe3a533d6f 42a4899821eb4624b542589b8f3c154f - - -]
Exception during message handling
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp Traceback (most recent call last):
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/openstack/common/rpc/amqp.py", line 440, in _process_data
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp **args)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/guestagent/datastore/postgresql/manager.py", line 78, in prepare
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp self.set_db_to_listen(context)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/guestagent/datastore/postgresql/service/config.py", line 84, in set_db_to_listen
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp version=self._get_psql_version(),
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/guestagent/datastore/postgresql/service/config.py", line 48, in _get_psql_version
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp out, err = pgutil.execute('psql', '--version', timeout=30)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/guestagent/datastore/postgresql/pgutil.py", line 31, in execute
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp "sudo", "-u", "postgres", *command, **kwargs
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/common/utils.py", line 217, in execute_with_timeout
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp return execute(*args, **kwargs)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/trove/openstack/common/processutils.py", line 200, in execute
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp cmd=sanitized_cmd)
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp Command: sudo -u postgres psql --version
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp Exit code: 1
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp Stdout: u''
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp Stderr: u'Error: /var/lib/postgresql/9.3/main is not accessible; please fix the directory permissions (/var/lib/postgresql/9.3/ should be world readable)\n'
2015-01-13 19:23:20.293 1331 TRACE trove.openstack.common.rpc.amqp

Here is the problem:

postgresql doesn't migrate the data, when backup_info is passed in.

Look at line number 74 in guestagent/datastore/postgresql/manager.py, class Manager, definition prepare.

postgresql should be handled this similarly to mysql.

Here is what mysql does in guestagent/datastore/mysql/manager.py (from line 128):

            if os.path.exists(mount_point):
                #rsync exiting data
                device.migrate_data(mount_point)
            #mount the volume
            device.mount(mount_point)
            LOG.debug("Mounted the volume.")
            app.start_mysql()
        if backup_info:
            self._perform_restore(backup_info, context,
                                  mount_point, app)

Here is what postgresql does in guestagent/datastore/postgresql/manager.py (from line 73):

            if os.path.exists(mount_point):
                if not backup_info:
                    device.migrate_data(mount_point)
            device.mount(mount_point)

Notice that the above 'if not backup_info' statement causes the data migration to be skipped.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/147288

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147288
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=3efef282eac61b5d0afb1a5199c1dc8453501a74
Submitter: Jenkins
Branch: master

commit 3efef282eac61b5d0afb1a5199c1dc8453501a74
Author: Petr Malik <email address hidden>
Date: Wed Jan 14 15:10:43 2015 -0500

    Trove create with --backup fails for postgresql

    Here are the steps to reproduce:
    1. Create a postgresql instance:

    trove create postgresql_instance 100 --datastore postgresql
    --datastore_version 9.3 --size 1 --databases db1 --users user1:password1

    2. Verify it is ACTIVE:

    trove list

    3. Create a backup of this new instance:

    trove backup-create <id> postgresql_instance-backup
    --description postgresql_instance-description

    4. Verify backup is complete:

    trove backup-list

    5. Create a new instance, using this backup instance:

    trove create postgresql_instance-restored 100 --datastore postgresql
    --datastore_version 9.3 --size 1 --backup <backup_id>

    Here are the problems:
    1. postgresql doesn't migrate the data, when backup_info is passed in.
    2. postgresql outputs few benign messages into the stderr stream
    during a normal restore procedure.

    Here is the solution:
    1. make sure that the postgresql manager always migrates the data by
    not skipping the that step when recreating an instance from
    backup in 'guestagent/datastore/postgresql/manager.py'.
    2. watch for the expected error messages in the stderr stream
    and avoid raising an exception in response.
    Message 'ERROR: role "postgres" already exists'
    is expected and does not pose any problems to the restore operation.
    We override the default error handling behavior
    in 'guestagent/strategies/restore/postgresql_impl.py'.

    UPDATES:
    - Applied suggested style improvements.
    - Use message whitelist rather than sequence.
    - Match messages by regex.

    Change-Id: Iac2ebd1ddb3a0131901e5b305d210c3c5b8feaea
    Closes-bug: 1410571