Abstract 'mv'/'cp' shell commands in guestagent

[Petr Malik]

Abstract 'mv'/'cp' shell commands in guestagent

see-also: bug/1431618

The goal of this patch set is to provide a simple straightforward
interface for most common shell calls used across the guestagent
code while encapsulating the related shell code into a single module.

This patch set includes refactoring of 'mv' and 'cp'
calls. It adds unit tests ('guestagent/test_operating_system')
for the above implementations.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169529

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/170642

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170642
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=2eb05f8897bb17def1573380da0a05942c42cf29
Submitter: Jenkins
Branch: master

commit 2eb05f8897bb17def1573380da0a05942c42cf29
Author: Petr Malik <email address hidden>
Date: Thu Apr 2 15:23:57 2015 -0400

    Abstract rm/chmod shell commands in guestagent

    The guestagent commonly executes basic shell commands
    across multiple files and datastores.
    Each datastore currently scripts its own shell calls.
    Apart from cluttering the Python code with shell constructs,
    it also makes maintenance of the shell calls more difficult by
    requiring the same change be made across multiple lines,
    source files and/or potentially even datastores.

    The goal of this patch set is to provide a simple straightforward
    interface for most common shell calls used across the guestagent
    code while encapsulating the related shell code into a single module.

    This patch set includes refactoring of 'rm' and 'chmod' calls.
    It also introduces unit tests ('guestagent/test_operating_system')
    for the above implementations.

    The shell code was moved into 'common/operating_system' module.
    The replacing interface is:

    def remove(path, force=False, recursive=True, **kwargs):
        """Remove a given file or directory.

    def chmod(path, mode, recursive=True, force=False, **kwargs):
        """Changes the mode of a given file.

    The calls also take optional keyword arguments:
        :param as_root: Execute as root.
        :type as_root: boolean

        :param timeout: Number of seconds if specified,
                                 default if not.
                                 There is no timeout if set to None.
        :type timeout: integer

    Notes on the representation of file permissions (modes):

    Generally, one can 'reset' (set modes to an exact value) or
    'add'/'remove' modes to/from the current mode.
    The above operations are represented by a new
    'operating_system.FileMode' class.
    For example, the read permission for all can be represented in the octal
    form as '444'.
    Mode on a given file can be 'reset' to this exact value
    (chmod =444 "path")
    The permissions can be added to (chmod +444 "path")
    ... or removed from the current mode (chmod -444 "path")

    The Modes constructor takes three optional lists of octal modes (flags),
    one for each of the above operations.

    "chmod u+x,g=rw,o=r" can be represented as:
    FileMode(reset=[0064], add=[0100])
    or
    FileMode(reset=[0040, 0020, 0004], add=[0100])
    or
    FileMode(reset=[0040 | 0020 | 0004], add=[0100])

    See 'operating_system.FileMode' struct for some common file modes.
    See the Python 'stat' module for more flags also accepted by the
    Python build-in functions such as 'os.chmod'.

    Change-Id: Idbfa05fdc5c48aed31989578f828a7f57aca3be2
    Closes-Bug: 1438429
    Related-Bug: 1431618
    Related-Bug: 1438430

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169529
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=4394396b233aa95dbb39b15148a61104e33f1fc3
Submitter: Jenkins
Branch: master

commit 4394396b233aa95dbb39b15148a61104e33f1fc3
Author: Petr Malik <email address hidden>
Date: Mon Mar 30 17:52:27 2015 -0400

    Abstract 'mv'/'cp' shell commands in guestagent

    The guestagent commonly executes basic shell commands
    across multiple files and datastores.
    Each datastore currently scripts its own shell calls.
    Apart from cluttering the Python code with shell constructs,
    it also makes maintenance of the shell calls more difficult by
    requiring the same change be made across multiple lines,
    source files and/or potentially even datastores.

    The goal of this patch set is to provide a simple straightforward
    interface for most common shell calls used across the guestagent
    code while encapsulating the related shell code into a single module.

    This patch set includes refactoring of 'mv' and 'cp' calls.
    It also introduces unit tests ('guestagent/test_operating_system')
    for the above implementations.

    The shell code was moved into 'common/operating_system' module.
    The replacing interface is:

    def move(source, destination, force=False, **kwargs):
        """Move a given file or directory to a new location.
        Move attempts to preserve the original ownership, permissions and
        timestamps.

    def copy(source, destination, force=False, preserve=False,
      recursive=True, **kwargs):
        """Copy a given file or directory to another location.
        Copy does NOT attempt to preserve ownership, permissions and timestamps
        unless the 'preserve' option is enabled.

    The calls also take optional keyword arguments:
        :param as_root: Execute as root.
        :type as_root: boolean

        :param timeout: Number of seconds if specified,
                                 default if not.
                                 There is no timeout if set to None.
        :type timeout: integer

    Change-Id: Ib969a111d102b817c9f18437c1f36bd17e3a8a4b
    Depends-On: Idbfa05fdc5c48aed31989578f828a7f57aca3be2
    Closes-Bug: 1438430
    Related-Bug: 1431618

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164034
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=0397f9c6a00c200572a55ba95d8a92a7a9596168
Submitter: Jenkins
Branch: master

commit 0397f9c6a00c200572a55ba95d8a92a7a9596168
Author: Jenkins <email address hidden>
Date: Thu Mar 12 10:20:51 2015 +0000

    Abstract 'mkdir' shell commands in guestagent

    The guestagent commonly executes basic shell commands
    such as 'mkdir' across multiple files and datastores.
    Each datastore currently scripts its own shell calls.
    Apart from cluttering the Python code with shell constructs,
    it also makes maintenance of the shell calls more difficult by
    requiring the same change be made across multiple lines,
    source files and/or potentially even datastores.

    The goal of this patch set is to provide a simple straightforward
    interface for most common shell calls used across the guestagent
    code while encapsulating the related shell code into a single module.

    This patch set includes refactoring of 'mkdir' and related 'chown'
    calls.
    It also introduces unit tests ('guestagent/test_operating_system')
    for the above implementations.

    The shell code was moved into 'common/operating_system' module.
    The replacing interface is:

    def create_directory(dir_path, user=None, group=None, force=True):
        """Create a given directory and update its ownership
        (recursively) to the given user and group if any."""

    The existing interface for 'chown' has been extended to maintain
    consistency with 'mkdir'.

    def chown(path, user, group, recursive=True, force=False):
        """Changes the owner and group of a given file.
        """

    The calls also take optional keyword arguments:
        :param as_root: Execute as root.
        :type as_root: boolean

        :param timeout: Number of seconds if specified,
                                 default if not.
                                 There is no timeout if set to None.
        :type timeout: integer

    Most occurrences of the above two shell calls ('mkdir' and 'chown')
    in the guestagent code have been replaced by the above abstraction.
    I did not replace native Python os.chown() and os.mkdir() calls
    and calls that are executed as a part of a larger shell script
    in this patch set.

    Authored-By: Petr Malik <email address hidden>
    Change-Id: I0fe96301087e3bd49d937c7c489006fd9d84ca1e
    Depends-On: Ib969a111d102b817c9f18437c1f36bd17e3a8a4b
    Closes-Bug: 1431618
    Related-Bug: 1438430