Delete mysql error log file as root on restore

Bug #1423759 reported by Petr Malik
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack DBaaS (Trove)
Fix Released
High
Petr Malik

Bug Description

Delete mysql error log file as root on restore

The Problem:
The current code let's Python manage the MySQL error log file generated
during the root password reset on backup restore (see bug/1419995).
The problem is that the database service may assume ownership of that
file in the process. The 'restricted deletion' policy dictates that
only owners (and the directory owner) can delete temporary files.
If this happens the cleanup step fails halting the whole
restore process.
NOTE: Not all datastores neccessarilly exhibit this behavior
(i.e. taking ownership of the log file).
It has been confirmed on Percona 5.5 running under Ubuntu Linux.

The Reproduction (important):
This issue is masked in the default testing (redstack) environment.
Redstack (in 'scripts/functions_qemu') sets 'GUEST_LOGDIR'
to the 'log_dir' value from 'etc/trove/trove-guestagent.conf.sample'
which happenns to be '/tmp/'.
Later in 'scripts/files/trove-guest.upstart.conf' assumes the ownership
of that directory: "chown GUEST_USERNAME:root GUEST_LOGDIR"
effectivelly hiding this issue.
The temporary directory would generally not be owned by the trove user
in a production environment.

In order to reproduce this issue in Redstack set 'GUEST_LOGDIR' to
some directory other than '/tmp', like '/var/log/trove/',
kick-start Percona 5.5 (remember to remove the cached
image if any) and run instance backup/restore or int-tests.

The Solution:
Do not let Trove do the cleanup on the error log file.
Force-delete the file as root, logging but not raising any exceptions.

Tags: mysql restore
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to trove (master)

Fix proposed to branch: master
Review: https://review.openstack.org/157972

Changed in trove:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to trove (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/157973

Changed in trove:
assignee: Petr Malik (pmalik) → Peter Stachowski (peterstac)
Changed in trove:
assignee: Peter Stachowski (peterstac) → nobody
Changed in trove:
assignee: nobody → Petr Malik (pmalik)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to trove (master)

Reviewed: https://review.openstack.org/157972
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=0ddd79ed3f9579f96541721991b7a4ea8d108997
Submitter: Jenkins
Branch: master

commit 0ddd79ed3f9579f96541721991b7a4ea8d108997
Author: Petr Malik <email address hidden>
Date: Wed Feb 18 18:16:20 2015 -0500

    Delete mysql error log file as root on restore

    The Problem:
    The current code let's Python manage the MySQL error log file generated
    during the root password reset on backup restore (see bug/1419995).
    The problem is that the database service may assume ownership of that
    file in the process. The 'restricted deletion' policy dictates that
    only owners (and the directory owner) can delete temporary files.
    If this happens the cleanup step fails halting the whole
    restore process.
    NOTE: Not all datastores necessarily exhibit this behavior
    (i.e. taking ownership of the log file).
    It has been confirmed on Percona 5.5 running under Ubuntu Linux.

    The Reproduction (important):
    This issue is masked in the default testing (redstack) environment.
    Redstack (in 'scripts/functions_qemu') sets 'GUEST_LOGDIR'
    to the 'log_dir' value from 'etc/trove/trove-guestagent.conf.sample'
    which happenns to be '/tmp/'.
    Later in 'scripts/files/trove-guest.upstart.conf' assumes the ownership
    of that directory: "chown GUEST_USERNAME:root GUEST_LOGDIR"
    effectively hiding this issue (see bug/1423760).
    The temporary directory would generally not be owned by the trove user
    in a production environment.

    In order to reproduce this issue in Redstack set 'GUEST_LOGDIR' to
    some directory other than '/tmp', like '/var/log/trove/',
    kick-start Percona 5.5 (remember to remove the cached
    image if any) and run instance backup/restore or int-tests.

    The Solution:
    Do not let Trove do the cleanup on the error log file.
    Force-delete the file as root, logging but not raising any exceptions.

    Tested on: MySQL 5.5/5.6 and Percona 5.5/5.6

    Change-Id: I9dd6ed543a01ecc4f84065ea4bf3737960de6e24
    Closes-Bug: 1423759
    Related-Bug: 1423760

Changed in trove:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to trove (master)

Reviewed: https://review.openstack.org/157973
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=004403c7b7c36c7e027494681b99223542df64ba
Submitter: Jenkins
Branch: master

commit 004403c7b7c36c7e027494681b99223542df64ba
Author: Petr Malik <email address hidden>
Date: Fri Feb 20 11:59:02 2015 -0500

    Do not use '/tmp' as default guestagent log location

    The Problem:
    Redstack sets 'GUEST_LOGDIR' to the 'log_dir' value from
    'etc/trove/trove-guestagent.conf.sample' which happens to be '/tmp/'.

    Aside from not being the canonical log file destination,
    temporary directory in Linux is a subject to the, so called,
    'restricted deletion' policy which dictates that only file owners
    (and the directory owner) can delete the files, irrespective of
    other access modifiers on the directory.

    Redstack changes the owner of 'GUEST_LOGDIR' (default='/tmp')
    to the 'trove' user. This may easily mask any potential issues with
    the 'restricted deletion' that would only show up later on
    production systems where '/tmp' is commonly owned by the root
    (see bug/1423759).

    The Solution:
    Change the default value of 'log_dir' to a directory
    which is not subject to the 'restricted deletion'.
    Chose '/var/log/trove/' as it is a common place for
    trove-related log files on the guestagent.

    Change-Id: I39d801a7e19f329c129a0c6df0c3987049d16394
    Closes-Bug: 1423760
    Related-Bug: 1423759
    Depends-On: I9dd6ed543a01ecc4f84065ea4bf3737960de6e24

Changed in trove:
milestone: none → kilo-3
Thierry Carrez (ttx)
Changed in trove:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in trove:
milestone: kilo-3 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.