fuel-mirror (version 8.0) create failed when http 302 found

[junkao]

Hi,

fuel-mirror(version 8.0) create failed when http 302 found

debug log

FO packetary.drivers loaded: 1077 packages from mos8.0.main.
[2016-03-19 05:16:39,009] DEBUG packetary.library start request: http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/dists/mos8.0/restricted/binary-amd64/Packages.gz
[2016-03-19 05:16:39,572] ERROR packetary.library request failed: http://mirror.fuel-infra.org/mos-repos/ubuntu/8.0/dists/mos8.0/restricted/binary-amd64/Packages.gz - 302(Found), retries left - 4.
[2016-03-19 05:16:39,573] DEBUG packetary.library start request: http://120.52.72.52:80/mirror.fuel-infra.org/c3pr90ntcsf0/mos-repos/ubuntu/8.0/dists/mos8.0/restricted/binary-amd64/Packages.gz
[2016-03-19 05:16:39,573] ERROR fuel-mirror offset
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 374, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 54, in run
    self.take_action(parsed_args)
  File "/usr/lib/python2.7/site-packages/fuel_mirror/commands/create.py", line 48, in take_action
    repos_reqs.get(group_name)
  File "/usr/lib/python2.7/site-packages/packetary/api.py", line 148, in clone_repositories
    packages = self._get_packages(repositories, debs, requirements)
  File "/usr/lib/python2.7/site-packages/packetary/api.py", line 209, in _get_packages
    main_index.add
  File "/usr/lib/python2.7/site-packages/packetary/controllers/repository.py", line 82, in load_packages
    self.driver.get_packages(connection, r, consumer)
  File "/usr/lib/python2.7/site-packages/packetary/drivers/deb_driver.py", line 119, in get_packages
    stream = GzipDecompress(connection.open_stream(index))
  File "/usr/lib/python2.7/site-packages/packetary/library/connections.py", line 212, in open_stream
    return self.opener.open(request)
  File "/usr/lib64/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 442, in error
    result = self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 629, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 402, in open
    req = meth(req)
  File "/usr/lib/python2.7/site-packages/packetary/library/connections.py", line 105, in http_request
    if request.offset > 0:
  File "/usr/lib64/python2.7/urllib2.py", line 229, in __getattr__
    raise AttributeError, attr
AttributeError: offset

[Dmitry Klenov (dklenov)]

@junkao, please specify the following info:
* Exact command with parameters you used
* Which repositories did you want to mirror
* Did you use proxy
* Content of fuel-createmirror configuration folder (/etc/fuel-createmirror)

[junkao (junkao)]

@Dmitry Klenov

* Exact command with parameters you used

fuel-createmirror -M

fuel-mirror create -G mos -I /usr/share/fuel-mirror/ubuntu.yaml --debug

fuel-mirror create -G ubuntu -I /usr/share/fuel-mirror/ubuntu.yaml --debug

* Which repositories did you want to mirror

Want to mirror mos

* Did you use proxy

no proxy

* Content of fuel-createmirror configuration folder (/etc/fuel-createmirror)

i can't find this folder in fuel 8.0 version

I can find the following directory

/usr/share/fuel-mirror/ubuntu.yaml
/usr/share/fuel-mirror/centos.yaml
/etc/fuel-mirror/config.yaml

======================

I think we need a function to process it when HTTP 302 redirect found

https://github.com/openstack/fuel-mirror/blob/stable/8.0/packetary/library/connections.py
def http_response(self, request, response):
        """Wraps response in a ResumableResponse.
        Checks that partial request completed successfully.
        :param request: the instance of RetryableRequest
        :param response: the response object
        :return: ResumableResponse if success otherwise same response
        """
        code, msg = response.getcode(), response.msg
        # the server should response partial content if range is specified

------------------>

         if code == 302:
              something....

<-----------------------

if request.offset > 0 and code != 206:
            raise RangeError(msg)

        if code >= 400:
            logger.error(
                "request failed: %s - %d(%s), retries left - %d.",
                request.get_full_url(), code, msg, request.retries_left - 1
            )
            if is_retryable_http_error(code) and request.retries_left > 0:
                time.sleep(request.get_retry_interval())
                request.retries_left -= 1
                response = self.parent.open(request)
            # pass response to next handler as is.
            return response

        logger.debug(
            "request completed: %s - %d (%s), duration - %d ms.",
            request.get_full_url(), response.getcode(), response.msg,
            int((time.time() - request.start_time) * 1000)
        )

        return ResumableResponse(request, response, self.parent)

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexey Shtokolov (ashtokolov)]

Doesn't affect >=9.0

[Bulat Gaifullin (bulat.gaifullin)]

The fuel-mirror will be replaced by packetary in fuel 10.0, so the bug is actual only for 9.0 and 8.0

[Rodion Tikunov (rtikunov)]

Reproduced in 8.0 and 9.0.

[Bulat Gaifullin (bulat.gaifullin)]

create linked bug in project packetary #1592464.
the fix will be proposed to packetary.

[Bulat Gaifullin (bulat.gaifullin)]

the bug is not high. because usually the http redirects is not used in repositories.
Fuel-mirror is deprecated project and should be replaced by packetary.