Re-write fuel-createmirror in python

[Vladimir Kozhukalov]

Currently it is barely maintainable. We need to totally re-design it and re-implement.

[Bulat Gaifullin (bulat.gaifullin)]

the script will be split to the following components:

file-transfer protocol
=================

incapsulates the protocols, that are supported to transfer file.

.. code:: python

  @six.add_metaclass(abc.ABCMeta)
  class Transport(object):
      @abc.abstractmethod
      def glob(self, path, mask=None):
          """enumerate files in patch
          returns list of pairs (isdir, name)
          """

      @abc.abstractmethod
      def exists(self, path):
          """checks that path exists"""

      @abc.abstractmethod
      def fetch(self, src, dst):
          """copies file from src to dst"""

protocols are planned to support: rsync, http[1]

repositories
==========

contains functional to work with repository

.. code:: python
   class Package:
          name # package name
          version = # package version
          requires = [] #list of required packages
          repository # the package repository

  @six.add_metaclass(abc.ABCMeta)
  class Repository(object):
      @abc.abstractmethod
      def get_packages(self, filter=None):
          """get list of all packages."""

      @abc.abstractmethod
      def clone(self, path):
          """clones repository to local fs"""

      @abc.abstractmethod
      def replicate(self, package, mirror):
          """replicates package to mirror"""

  @six.add_metaclass(abc.ABCMeta)
class Mirror(object):
      @abc.abstractmethod
       def get_path(package):
            """get path for package"""

      @abc.abstractmethod
       def is_dirty(package):
            """checks that re-sync is required"""

repositories are planned to support: ubuntu - repository, yum repositories[1]

modules:

command
========

provides function to launch sub-commands and dry-run semantic

.. code:: python

def call(cmd, **kwargs):
   """for details see subprocess.call."""

def check_call(cmd, **kwargs):
   """for details see subprocess.call."""

def check_output(cmd, **kwargs):
   """for details see subprocess.call."""

def find_executables(executable):
   """get the fullpath for executable"""

createmirror:

.. code:: python

  def resolve(requires, repositories, dependencies):
        """resolves list of requires recursively."""

workflow:
  repositories = []
  for r in config.repositories:
     repositories.append(open_repository(r))

 packages = defaultdict() # repository -> set of packages

 for r in config.target_repository:
        for package in r.get_packages():
              # it is possible to start synchronisation mirror in parallel
              packages[r].add(package)
              resolve_requires(package.requires, repositories, packages)

local_repositories = []

for repository, packages in packages:
    mirror = repository.clone(config.target)
   # this part of code can be run in parallel
    for p in packages:
        repository.replicate(package, mirror)
   mirror.update()e
   local_repositories.append(mirror.get_path)

# dump information about local repositories.
# update fuel-main if required

configuration file:

.. code:: yaml

commands:
  dry_run: False
  retries: 1
  retry_delay: 1

logging:
  filename: /var/log/fuel-mirror.log
  level: debug

transport:
   ...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/231979

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234238

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/236079

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/236079
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=843736e50534d184f4b07744d68d0f2d27284cae
Submitter: Jenkins
Branch: master

commit 843736e50534d184f4b07744d68d0f2d27284cae
Author: Sergey Kulanov <email address hidden>
Date: Fri Oct 16 23:10:20 2015 +0300

    Fix fuel-createmirror spec

    * Since we are in migration stage to python code implementation
    we need to fix SPEC files both for RPM and DEB

    * Small lintian fixes

    Change-Id: I51e4a6aa27b2855805beab3c61b4d4c499df2732
    Related-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237630

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237640

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/231979
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=b6d09c7eade02dafbfe1bc011dd0e4ae86a0492b
Submitter: Jenkins
Branch: master

commit b6d09c7eade02dafbfe1bc011dd0e4ae86a0492b
Author: Bulat Gaifullin <email address hidden>
Date: Wed Oct 7 16:09:24 2015 +0300

    [packetary] initial commit

    the project structure was created by cookiecutter.

    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

    Change-Id: Ib33955ba1d4013bc1491a503610885367924b0e9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/237966

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238433

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238522

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238863

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bulat Gaifullin (<email address hidden>) on branch: master
Review: https://review.openstack.org/238863
Reason: obsoletes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/239657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/239657
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=5b9488899f9eb9f60a40f177b71a617aa7d50d4d
Submitter: Jenkins
Branch: master

commit 5b9488899f9eb9f60a40f177b71a617aa7d50d4d
Author: Bulat Gaifullin <email address hidden>
Date: Tue Oct 27 15:42:54 2015 +0300

    CI fix issues

    * Fix module import in packetary tests
    * Fix PEP8 in perestroika code

    Change-Id: I0a9fc3cbbe5a59152f3733db235c23a242c58b04
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237630
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=56689ae7cf202b9bdb9d39e5d7562548445bafc3
Submitter: Jenkins
Branch: master

commit 56689ae7cf202b9bdb9d39e5d7562548445bafc3
Author: Bulat Gaifullin <email address hidden>
Date: Tue Oct 20 18:08:10 2015 +0300

    [packetary] Infrastructure

    Introduce infrastructure:
    * Executor (The Asynchronous task executor)
    * ConnectionsPool (The pool of network connections)
    * ResumeableStream - allows resume streaming when error occures

    Change-Id: Ia68d60c2b9d685820a4c1916c8f1aa724f3a3f91
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237640
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=6a9c4630424bf2d410604d4c96a77fbf0cf0f9a4
Submitter: Jenkins
Branch: master

commit 6a9c4630424bf2d410604d4c96a77fbf0cf0f9a4
Author: Bulat Gaifullin <email address hidden>
Date: Tue Oct 20 18:33:58 2015 +0300

    [packetary] Packages indexer

    Change-Id: I981fe7ae8f0aba8314475c3633f325e18d5e3bb5
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/237966
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=671af8e61112cf8d90ba861364902c6b1c494b43
Submitter: Jenkins
Branch: master

commit 671af8e61112cf8d90ba861364902c6b1c494b43
Author: Bulat Gaifullin <email address hidden>
Date: Wed Oct 21 12:58:54 2015 +0300

    [packetary] Repository class

    class Repository composes from:
    * RepositryDriver - low-level support for physical repository. deb, yum, etc.
    * RepositoryController - infrastcuture method to communicate with driver
    * RepositoryApi - high-level class, that provides methods to work with repository

    Change-Id: Iaf868fca982d91089e369d13a6fb381ff879ea73
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/244151

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238433
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=aa570b1f98b12fd0bb7875e5f2e5544db6dfc523
Submitter: Jenkins
Branch: master

commit aa570b1f98b12fd0bb7875e5f2e5544db6dfc523
Author: Bulat Gaifullin <email address hidden>
Date: Tue Oct 20 18:33:58 2015 +0300

    [packetary] Introduce driver for debian-repository

    Change-Id: Icb8f6964bd0a9080ace50b4786ac76ae5dbb9976
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238522
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=994fed9b1ed889718b61a59733275c08c2dd4c64
Submitter: Jenkins
Branch: master

commit 994fed9b1ed889718b61a59733275c08c2dd4c64
Author: Bulat Gaifullin <email address hidden>
Date: Tue Oct 20 18:33:58 2015 +0300

    [packetary] Introduce driver for rpm-repository

    Change-Id: Ib9b9208964dba25d585fb9345a6ef95e7782a9ff
    Implements: blueprint refactor-local-mirror-scripts
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244151
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=d0b144b0b21cc3fd316f360518695a99d79e60b1
Submitter: Jenkins
Branch: master

commit d0b144b0b21cc3fd316f360518695a99d79e60b1
Author: Bulat Gaifullin <email address hidden>
Date: Wed Nov 11 18:10:45 2015 +0300

    Introduced fuel-mirror

    Avaliable commands:
    - create - Creates a new local mirrors.
      fuel-mirror create --input ubuntu.yaml

    - apply - Applies local mirrors for Fuel-environments.
      fuel-mirror apply --input centos.yaml -e "ENV1"

    Change-Id: I6cbf4e581a93b47cf75fb3c89aa1020a352784c8
    Closes-Bug: #1487077
    Implements: blueprint refactor-local-mirror-scripts

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234238
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=70db939a91e6569731b84433b87775550073e6d3
Submitter: Jenkins
Branch: master

commit 70db939a91e6569731b84433b87775550073e6d3
Author: Alexandr Kostrikov <email address hidden>
Date: Tue Oct 13 17:18:12 2015 +0300

    Tests to cover create mirror

    This is a tests to run on ci to
    verify fuel-createmirror behaviour.
    Tests to check network issues are skipped
    because of need to choose right network
    tool.

    Change-Id: I50179570b215dd26eaa5a5b0db339f69ad405716
    Partial-bug: #1487077

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/250485

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/250532

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/250642

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bulat Gaifullin (<email address hidden>) on branch: master
Review: https://review.openstack.org/250532

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/252254

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/252254
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=0d32e5472b1bcf56fbcdf7cc7eb2bab85031d1dd
Submitter: Jenkins
Branch: master

commit 0d32e5472b1bcf56fbcdf7cc7eb2bab85031d1dd
Author: bgaifullin <email address hidden>
Date: Wed Dec 2 11:32:26 2015 +0300

    Fixed fuel_mirror_commit uses for network_checker repo

    Change-Id: I7b3762c71ec4b0c40699dc4d97fdc3d3eda9b2f7
    Partial-Bug: #1487077

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: master
Change author: Sergey Kulanov <email address hidden>
Review: https://review.fuel-infra.org/14373

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/14373
Submitter: Vitaly Parakhin <email address hidden>
Branch: master

Commit: d9ea68cbaad2b9221b5c7c9f1ee879fbfa269ee6
Author: Sergey Kulanov <email address hidden>
Date: Wed Dec 2 14:09:22 2015

Build deb packages for python-fuelclient

Patchset-example: https://review.openstack.org/#/c/252303/

Change-Id: I64513ac8a6feee42f73a4de680b7e94b12ea162c
Related-Bug: #1487077
Related-Bug: #1489506

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250642
Committed: https://git.openstack.org/cgit/openstack/fuel-main/commit/?id=d1cda9fa50cb8f90e7655ee1046802c56ed3d69f
Submitter: Jenkins
Branch: master

commit d1cda9fa50cb8f90e7655ee1046802c56ed3d69f
Author: bgaifullin <email address hidden>
Date: Fri Nov 27 00:24:22 2015 +0300

    fuel-createmirror is replaced by fuel-mirror

    fuel-package-updates is also replaced by fuel-mirror

    Depends-On: #I296a5f356ca805deda28ea984df15c0b63b43a80
    Closes-Bug: #1487077
    Implements: blueprint refactor-local-mirror-scripts
    Doc-Impact

    Change-Id: I72e9ced2c20c615e841ec73b9c7e7bca022918cf

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250485
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=2e3999586445ebd9bc8fea2571f7159d41034ff5
Submitter: Jenkins
Branch: master

commit 2e3999586445ebd9bc8fea2571f7159d41034ff5
Author: Bulat Gaifullin <email address hidden>
Date: Thu Nov 26 19:34:32 2015 +0300

    Added packaging for fuel-mirror

    Change-Id: I296a5f356ca805deda28ea984df15c0b63b43a80
    Closes-Bug: #1487077
    Closes-Bug: #1489506
    Implements: blueprint refactor-local-mirror-scripts

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/253836

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/244137
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=af0a66d31cfb9011d427e4758e0f61a1aa804180
Submitter: Jenkins
Branch: master

commit af0a66d31cfb9011d427e4758e0f61a1aa804180
Author: Bulat Gaifullin <email address hidden>
Date: Wed Nov 11 17:35:28 2015 +0300

    [packetary] Introduce command-line interface

    Available commands:
    - clone - Clones specified repository(es) to local folder.
      packetary clone -o "http://archive.ubuntu.com/ubuntu trusty main restricted universe multiverse" -t deb -d /var/www/mirror/ubuntu

    - packages - Gets the list of packages from repository(es).
      packetary packages -o http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/os -t yum

    - unresolved - Get the list of external dependencies for repository.
      packetary unresolved -o "http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0 mos8.0 main"

    Change-Id: I4b3f3fb7b7f2967cf596ed3c7758cfbbf76dfe73
    Partial-Bug: #1487077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/253836
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=7d429bdd3f1e3bc6524de6c67723fa010a31a52f
Submitter: Jenkins
Branch: master

commit 7d429bdd3f1e3bc6524de6c67723fa010a31a52f
Author: Bulat Gaifullin <email address hidden>
Date: Sat Dec 5 22:20:35 2015 +0300

    Removed obsolete version of fuel-createmirror

    fuel-createmirror was replaced by fuel-mirror

    Change-Id: I336fac39c8b1dd0176cbd1df044e9e1860ad8fb6
    Related-Bug: #1487077

[Tatyanka (tatyana-leontovich)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "466"
  build_id: "466"
  fuel-nailgun_sha: "f81311bbd6fee2665e3f96dcac55f72889b2f38c"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "6823f1d4005a634b8436109ab741a2194e2d32e0"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "fe03d887361eb80232e9914eae5b8d54304df781"
  fuel-ostf_sha: "ab5fd151fc6c1aa0b35bc2023631b1f4836ecd61"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "fac143f4dfa75785758e72afbdc029693e94ff2b"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "727f7076f04cb0caccc9f305b149a2b5b5c2af3a"