fuel-mirror does not handle redirects

[Michal Skalski]

Detailed bug description:

When remote server respond with 302 HTTP during fetching packages fuel-mirror will exit with error:

http://paste.openstack.org/show/516803/

Fuel-mirror is using standard urllib python library but also extend request class by adding new attributes [0]. When one of the 30x HTTP codes are returned urllib internally recreate request object [1] which dose not contain extra attributes like offset which lead to exception when we try access non-existent attribute[2].

[0] https://github.com/openstack/fuel-mirror/blob/stable/mitaka/packetary/library/connections.py#L42
[1] https://github.com/python-git/python/blob/master/Lib/urllib2.py#L543-L546
[2] https://github.com/openstack/fuel-mirror/blob/stable/mitaka/packetary/library/connections.py#L107

Steps to reproduce:

In fuel mirror configuration use address of mirror server which return HTTP 302 Code when requesting packages. Example of such repository is: http://ftp.acc.umu.se/ubuntu

mirror-config.yaml: http://paste.openstack.org/show/516805/
ubuntu.yaml: http://paste.openstack.org/show/516807/

Please take in to account that above configuration is for fuel-9.0

fuel-mirror --debug --config ./mirror-config.yaml create --group ubuntu --pattern=ubuntu

Expected results:

fuel-mirror should create local mirror without error.

Actual result:

fuel-mirror exit before finishing because Too many errors occurred.

Reproducibility:

With mirror http://ftp.acc.umu.se/ubuntu I was able to reproduce this every time.

Workaround:

Based on my experience not all mirror servers return 302 HTTP code, changing url of the mirror from which we download packages can help.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/331041

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/333916

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michal Skalski (<email address hidden>) on branch: master
Review: https://review.openstack.org/331041
Reason: I will back port Bulat change: https://review.openstack.org/#/c/330049/
to have common solution in packetary and fuel-mirror.

New change: https://review.openstack.org/#/c/333916/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333916
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=192a3d9f8f993afb12c5108dd9339c6688c23e11
Submitter: Jenkins
Branch: master

commit 192a3d9f8f993afb12c5108dd9339c6688c23e11
Author: Bulat Gaifullin <email address hidden>
Date: Fri Jun 24 15:52:41 2016 +0200

    Fixed handling http redirects

    Override method 'redirect_request' for patching
    new request, that has been created on handling
    http redirect.

    Change-Id: I40db406e2377bebec1113639b91a0b5262e2e9ad
    Closes-Bug: 1593674

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/334882

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/334882
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=1de3a40c204417377ba6aa4f5bc780a4332a025b
Submitter: Jenkins
Branch: stable/mitaka

commit 1de3a40c204417377ba6aa4f5bc780a4332a025b
Author: Bulat Gaifullin <email address hidden>
Date: Fri Jun 24 15:52:41 2016 +0200

    Fixed handling http redirects

    Override method 'redirect_request' for patching
    new request, that has been created on handling
    http redirect.

    Change-Id: I40db406e2377bebec1113639b91a0b5262e2e9ad
    Closes-Bug: 1593674
    (cherry picked from commit 192a3d9f8f993afb12c5108dd9339c6688c23e11)

[Ekaterina Shutova (eshutova)]

Verified on 9.2 snapshot #798.