Re-write fuel-createmirror in python
Bug #1487077 reported by
Vladimir Kozhukalov
This bug affects 2 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Bulat Gaifullin |
Bug Description
Currently it is barely maintainable. We need to totally re-design it and re-implement.
Changed in fuel: | |
assignee: | Fuel build team (fuel-build) → Bulat Gaifullin (bgaifullin) |
tags: | added: feature |
Changed in fuel: | |
status: | Confirmed → Triaged |
Changed in fuel: | |
status: | Triaged → In Progress |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexey Shtokolov (ashtokolov) |
Changed in fuel: | |
assignee: | Alexey Shtokolov (ashtokolov) → Bulat Gaifullin (bgaifullin) |
tags: | added: area-python |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexey Shtokolov (ashtokolov) |
Changed in fuel: | |
assignee: | Alexey Shtokolov (ashtokolov) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
Changed in fuel: | |
assignee: | Bulat Gaifullin (bgaifullin) → Alexandr Kostrikov (akostrikov-mirantis) |
Changed in fuel: | |
assignee: | Alexandr Kostrikov (akostrikov-mirantis) → Bulat Gaifullin (bgaifullin) |
tags: | added: on-verification |
To post a comment you must log in.
the script will be split to the following components:
file-transfer protocol
=================
incapsulates the protocols, that are supported to transfer file.
.. code:: python
@six. add_metaclass( abc.ABCMeta) abc.abstractmet hod
"""enumerate files in patch
class Transport(object):
@
def glob(self, path, mask=None):
returns list of pairs (isdir, name)
"""
@ abc.abstractmet hod
def exists(self, path):
"""checks that path exists"""
@ abc.abstractmet hod
def fetch(self, src, dst):
"""copies file from src to dst"""
protocols are planned to support: rsync, http[1]
repositories
==========
contains functional to work with repository
.. code:: python
repository # the package repository
class Package:
name # package name
version = # package version
requires = [] #list of required packages
@six. add_metaclass( abc.ABCMeta) abc.abstractmet hod
class Repository(object):
@
def get_packages(self, filter=None):
"""get list of all packages."""
@ abc.abstractmet hod
def clone(self, path):
"""clones repository to local fs"""
@ abc.abstractmet hod
"""replicate s package to mirror"""
def replicate(self, package, mirror):
@six. add_metaclass( abc.ABCMeta) abc.abstractmet hod
class Mirror(object):
@
def get_path(package):
"""get path for package"""
@ abc.abstractmet hod
"" "checks that re-sync is required"""
def is_dirty(package):
repositories are planned to support: ubuntu - repository, yum repositories[1]
modules:
command
========
provides function to launch sub-commands and dry-run semantic
.. code:: python
def call(cmd, **kwargs):
"""for details see subprocess.call."""
def check_call(cmd, **kwargs):
"""for details see subprocess.call."""
def check_output(cmd, **kwargs):
"""for details see subprocess.call."""
def find_executable s(executable) :
"""get the fullpath for executable"""
createmirror:
.. code:: python
def resolve(requires, repositories, dependencies):
"""resolves list of requires recursively."""
workflow: repositories: es.append( open_repository (r))
repositories = []
for r in config.
repositori
packages = defaultdict() # repository -> set of packages
for r in config. target_ repository:
packages[ r].add( package)
resolve_ requires( package. requires, repositories, packages)
for package in r.get_packages():
# it is possible to start synchronisation mirror in parallel
local_repositories = []
for repository, packages in packages: clone(config. target)
repository. replicate( package, mirror) repositories. append( mirror. get_path)
mirror = repository.
# this part of code can be run in parallel
for p in packages:
mirror.update()e
local_
# dump information about local repositories.
# update fuel-main if required
configuration file:
.. code:: yaml
commands:
dry_run: False
retries: 1
retry_delay: 1
logging: fuel-mirror. log
filename: /var/log/
level: debug
transport:
...