Create a local mirror failed for http://mirror.fuel-infra.org/robots.txt: ERROR 404: Not Found.

[lidiwen]

Create local mirror with Fuel CLI, it always stuck, run the wget command separately, it shows "http://mirror.fuel-infra.org/robots.txt:
2015-10-30 07:00:02 ERROR 404: Not Found."

[root@fuel docker-logs]# fuel-package-updates -d centos -r 2014.2.2-6.1 -v -s [my fuel master node IP] -p [my fuel master node password]

2015-10-30 06:58:41,374 - DEBUG - Using http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/ as mirror URL.
2015-10-30 06:58:41,374 - INFO - Started mirroring remote repository...
2015-10-30 06:58:41,374 - DEBUG - Execute command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2014.2.2-6.1/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/"
2015-10-30 06:58:41,374 - DEBUG - Execute command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2014.2.2-6.1/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/"
2015-10-30 06:58:41,376 - DEBUG - Stdout and stderr of command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2014.2.2-6.1/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/":
^CTraceback (most recent call last):
  File "/usr/bin/fuel-package-updates", line 9, in <module>
    load_entry_point('fuel-package-updates==6.0.0', 'console_scripts', 'fuel-package-updates')()
  File "/usr/lib/python2.6/site-packages/fuel_package_updates/fuel_package_updates.py", line 661, in main
    settings.exclude_dirs, options.distro)
  File "/usr/lib/python2.6/site-packages/fuel_package_updates/fuel_package_updates.py", line 553, in mirror_remote_repository
    if exec_cmd(download_cmd) != 0:
  File "/usr/lib/python2.6/site-packages/fuel_package_updates/fuel_package_updates.py", line 335, in exec_cmd
    logger.debug(line.rstrip())
KeyboardInterrupt

[root@fuel ~]# wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2014.2.2-6.1/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/
Last-modified header missing -- time-stamps turned off.
2015-10-30 07:00:02 URL:http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/ [548] -> "/var/www/nailgun/2014.2.2-6.1/centos/updates/index.html" [1]
http://mirror.fuel-infra.org/robots.txt:
2015-10-30 07:00:02 ERROR 404: Not Found.
Last-modified header missing -- time-stamps turned off.
2015-10-30 07:00:03 URL:http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/Packages/ [16053] -> "/var/www/nailgun/2014.2.2-6.1/centos/updates/Packages/index.html" [1]
Last-modified header missing -- time-stamps turned off.
2015-10-30 07:00:04 URL:http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/Sources/ [3314] -> "/var/www/nailgun/2014.2.2-6.1/centos/updates/Sources/index.html" [1]
Last-modified header missing -- time-stamps turned off.
2015-10-30 07:00:04 URL:http://mirror.fuel-infra.org/mos/centos-6/mos6.1/updates/repodata/ [697] -> "/var/www/nailgun/2014.2.2-6.1/centos/updates/repodata/index.html" [1]

[root@fuel ~]# fuel --fuel-version
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
api: '1.0'
astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
auth_required: true
build_id: 2015-06-19_13-02-31
build_number: '525'
feature_groups:
- mirantis
fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
openstack_version: 2014.2.2-6.1
production: docker
python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
release: '6.1'
release_versions:
  2014.2.2-6.1:
    VERSION:
      api: '1.0'
      astute_sha: 1ea8017fe8889413706d543a5b9f557f5414beae
      build_id: 2015-06-19_13-02-31
      build_number: '525'
      feature_groups:
      - mirantis
      fuel-library_sha: 2e7a08ad9792c700ebf08ce87f4867df36aa9fab
      fuel-ostf_sha: 8fefcf7c4649370f00847cc309c24f0b62de718d
      fuelmain_sha: a3998372183468f56019c8ce21aa8bb81fee0c2f
      nailgun_sha: dbd54158812033dd8cfd7e60c3f6650f18013a37
      openstack_version: 2014.2.2-6.1
      production: docker
      python-fuelclient_sha: 4fc55db0265bbf39c369df398b9dc7d6469ba13b
      release: '6.1'

[Maciej Relewicz (rlu)]

please attach diagnostic snapshot of your environment

[lidiwen (lidiwencs)]

diagnositic snapshot is so big, is uploading.
I find no robots.txt on "http://mirror.fuel-infra.org/robots.txt:", why does it need to find this file during create local mirror?

[Bulat Gaifullin (bulat.gaifullin)]

Seem like the problem in 'fuel_package_update.exec_cmd'. The execution hangs on this call.
It would be better to rework this function via subprocess.check_output or need to call Popen.communicate.

[Andrey Tykhonov (atykhonov)]

I'm not able to reproduce it with 8.0

tkhno@mts ~ $ ssh root@10.109.30.2

[root@nailgun ~]# fuel-package-updates -d centos -r 2014.2.2-6.1 -v -s 10.109.30.2 -p admin
2015-11-12 17:09:03,805 - DEBUG - Initiate Nailgun client with url http://10.109.30.2:8000
2015-11-12 17:09:03,805 - DEBUG - Initiate HTTPClient with url http://10.109.30.2:8000
2015-11-12 17:09:03,807 - WARNING - Authorization failure: Authentication required
2015-11-12 17:09:03,807 - DEBUG - Initialize keystoneclient with url http://10.109.30.2:5000/v2.0
2015-11-12 17:09:03,908 - DEBUG - Authorization token is successfully updated
2015-11-12 17:09:03,909 - DEBUG - Set X-Auth-Token to a973e74f08e84be5be157d4aec23c69e
Traceback (most recent call last):
  File "/usr/bin/fuel-package-updates", line 9, in <module>
    load_entry_point('fuel-package-updates==8.0.0', 'console_scripts', 'fuel-package-updates')()
  File "/usr/lib/python2.6/site-packages/fuel_package_updates/fuel_package_updates.py", line 687, in main
    options.release, ', '.join(supported_releases)))
fuel_package_updates.fuel_package_updates.UpdatePackagesException: Fuel release "2014.2.2-6.1" is not supported. Please specify one of the following: "2015.1.0-8.0, 2015.1.0-8.0". See help (--help) for details.

[root@nailgun ~]# fuel-package-updates -d centos -r 2015.1.0-8.0 -v -s 10.109.30.2 -p admin
2015-11-12 17:09:30,718 - DEBUG - Initiate Nailgun client with url http://10.109.30.2:8000
2015-11-12 17:09:30,718 - DEBUG - Initiate HTTPClient with url http://10.109.30.2:8000
2015-11-12 17:09:30,721 - WARNING - Authorization failure: Authentication required
2015-11-12 17:09:30,721 - DEBUG - Initialize keystoneclient with url http://10.109.30.2:5000/v2.0
2015-11-12 17:09:30,838 - DEBUG - Authorization token is successfully updated
2015-11-12 17:09:30,838 - DEBUG - Set X-Auth-Token to dc1c4fcef98c4ace97397e6ddb2c0865
2015-11-12 17:09:30,892 - DEBUG - Using http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/updates/ as mirror URL.
2015-11-12 17:09:30,908 - INFO - Started mirroring remote repository...
2015-11-12 17:09:30,909 - DEBUG - Execute command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2015.1.0-8.0/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/updates/"
2015-11-12 17:09:30,909 - DEBUG - Execute command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2015.1.0-8.0/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6-fuel/updates/"
2015-11-12 17:09:30,912 - DEBUG - Stdout and stderr of command "wget -N --recursive --no-parent --no-verbose -R "*.html" -R "*.gif" -R "*.key" -R "*.gpg" -R "*.dsc" -R "*.tar.gz" --exclude-directories='ubuntu/dists/mos?.?/,repodata' --directory-prefix /var/www/nailgun/2015.1.0-8.0/centos/updates -nH --cut-dirs=4 http://mirror.fuel-infra.org/mos-repos/centos/mos8.0-centos6...

[Bulat Gaifullin (bulat.gaifullin)]

fuel-package-updates was deprecated by fuel-mirror utility.