fuel-createmirror - doesn't work when created a partial and then full mirror

[Serhii Ovsianikov]

Input data

[root@fuel ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "525"
  build_id: "2015-06-19_13-02-31"
  nailgun_sha: "dbd54158812033dd8cfd7e60c3f6650f18013a37"
  python-fuelclient_sha: "4fc55db0265bbf39c369df398b9dc7d6469ba13b"
  astute_sha: "1ea8017fe8889413706d543a5b9f557f5414beae"
  fuel-library_sha: "2e7a08ad9792c700ebf08ce87f4867df36aa9fab"
  fuel-ostf_sha: "8fefcf7c4649370f00847cc309c24f0b62de718d"
  fuelmain_sha: "a3998372183468f56019c8ce21aa8bb81fee0c2f"

Env: Ubuntu, HA, Neutron + Vlan

On every nodes created the /etc/apt/sources.list.d/my.list file includes the following lines
deb http://10.30.0.2:8080/ubuntu-part/ trusty main universe multiverse
deb http://10.30.0.2:8080/ubuntu-part/ trusty-updates main universe multiverse
deb http://10.30.0.2:8080/ubuntu-part/ trusty-security main universe multiverse

Steps to reproduce
1. Creating a partial mirror of a repository (run the script fuel-createmirror)
2. Creating a full mirror of a repository
- change the variable PARTIAL_UPSTREAM=0 in the common.cfg configuration file
- run the script fuel-createmirror again
- wait until packages will be downloaded
3. Run the command apt-get update on a node
3. Install package (for example collectd) on node from a local repository which located on the Fuel master node

Expected result: a package will installed successfully
Actual result: we have lots of error
Err http://10.30.0.2:8080/ubuntu-part/ trusty/main hicolor-icon-theme all 0.13-1 404 Not Found
E: Failed to fetch http://10.30.0.2:8080/ubuntu-part/pool/main/a/alsa-lib/libasound2-data_1.0.27.2-3ubuntu7_all.deb 404 Not Found
...

The next files attached to the bug:
- all files from the directory /opt/fuel-createmirror-6.1/
- a log file /var/log/mirror-sync/ubuntu-update.log

root@node-3:/etc/apt/sources.list.d# apt-get install collectd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  cloud-guest-utils eatmydata python-configobj python-oauth python3-pycurl python3-software-properties software-properties-common unattended-upgrades
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  at-spi2-core ca-certificates-java collectd-core colord dconf-gsettings-backend dconf-service default-jre-headless fontconfig fontconfig-config fonts-dejavu fonts-dejavu-core fonts-dejavu-extra
  hicolor-icon-theme java-common libasound2 libasound2-data libasyncns0 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3
  libcairo-gobject2 libcairo2 libcanberra-gtk3-0 libcanberra-gtk3-module libcanberra0 libcolord1 libcolorhug1 libcups2 libdatrie1 libdbi1 libdconf1 libexif12 libflac8 libfontconfig1 libgd3
  libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgphoto2-6 libgphoto2-l10n libgphoto2-port10 libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common libgudev-1.0-0 libgusb2 libharfbuzz0b libieee1284-3
  libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 liblvm2app2.2 libmemcached10 libmodbus5 libnotify4 libogg0 liboping0 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpcsclite1
  libpixman-1-0 libpolkit-agent-1-0 libpolkit-backend-1-0 libpq5 libprotobuf-c0 libpulse0 librabbitmq1 librrd4 libsane libsane-common libsndfile1 libtdb1 libthai-data libthai0 libtiff5 libtokyocabinet9
  libtokyotyrant3 libupsclient3 libv4l-0 libv4lconvert0 libvarnishapi1 libvirt0 libvorbis0a libvorbisenc2 libvorbisfile3 libvpx1 libwayland-client0 libwayland-cursor0 libxcb-render0 libxcb-shm0
  libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxpm4 libxrandr2 libxrender1 libxtst6 libyajl2 notification-daemon openjdk-7-jre-headless policykit-1 rrdtool
  sound-theme-freedesktop ttf-dejavu ttf-dejavu-core ttf-dejavu-extra tzdata tzdata-java x11-common
Suggested packages:
  collectd-dev librrds-perl liburi-perl libhtml-parser-perl libregexp-common-perl apcupsd bind9 hddtemp ipvsadm lm-sensors mbmon nginx nut openvpn olsrd pdns-server postgresql time-daemon default-jre
  equivs libasound2-plugins alsa-utils libcanberra-gtk0 libcanberra-pulse cups-common libgd-tools gphoto2 gtkam librsvg2-common gvfs libjasper-runtime liblcms2-utils ttf-baekmuk ttf-arphic-gbsn00lp
  ttf-arphic-bsmi00lp ttf-arphic-gkai00mp ttf-arphic-bkai00mp pcscd pulseaudio avahi-daemon hplip hpoj libsane-extras sane-utils icedtea-7-jre-jamvm libnss-mdns sun-java6-fonts fonts-ipafont-gothic
  fonts-ipafont-mincho ttf-wqy-microhei ttf-wqy-zenhei ttf-indic-fonts-core ttf-telugu-fonts ttf-oriya-fonts ttf-kannada-fonts ttf-bengali-fonts
The following NEW packages will be installed:
  at-spi2-core ca-certificates-java collectd collectd-core colord dconf-gsettings-backend dconf-service default-jre-headless fontconfig fontconfig-config fonts-dejavu fonts-dejavu-core fonts-dejavu-extra
  hicolor-icon-theme java-common libasound2 libasound2-data libasyncns0 libatk-bridge2.0-0 libatk1.0-0 libatk1.0-data libatspi2.0-0 libavahi-client3 libavahi-common-data libavahi-common3
  libcairo-gobject2 libcairo2 libcanberra-gtk3-0 libcanberra-gtk3-module libcanberra0 libcolord1 libcolorhug1 libcups2 libdatrie1 libdbi1 libdconf1 libexif12 libflac8 libfontconfig1 libgd3
  libgdk-pixbuf2.0-0 libgdk-pixbuf2.0-common libgphoto2-6 libgphoto2-l10n libgphoto2-port10 libgraphite2-3 libgtk-3-0 libgtk-3-bin libgtk-3-common libgudev-1.0-0 libgusb2 libharfbuzz0b libieee1284-3
  libjasper1 libjbig0 libjpeg-turbo8 libjpeg8 liblcms2-2 liblvm2app2.2 libmemcached10 libmodbus5 libnotify4 libogg0 liboping0 libpango-1.0-0 libpangocairo-1.0-0 libpangoft2-1.0-0 libpcsclite1
  libpixman-1-0 libpolkit-agent-1-0 libpolkit-backend-1-0 libpq5 libprotobuf-c0 libpulse0 librabbitmq1 librrd4 libsane libsane-common libsndfile1 libtdb1 libthai-data libthai0 libtiff5 libtokyocabinet9
  libtokyotyrant3 libupsclient3 libv4l-0 libv4lconvert0 libvarnishapi1 libvirt0 libvorbis0a libvorbisenc2 libvorbisfile3 libvpx1 libwayland-client0 libwayland-cursor0 libxcb-render0 libxcb-shm0
  libxcomposite1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon0 libxpm4 libxrandr2 libxrender1 libxtst6 libyajl2 notification-daemon openjdk-7-jre-headless policykit-1 rrdtool
  sound-theme-freedesktop ttf-dejavu ttf-dejavu-core ttf-dejavu-extra tzdata-java x11-common
The following packages will be upgraded:
  tzdata
1 upgraded, 120 newly installed, 0 to remove and 87 not upgraded.
Need to get 57.2 MB of archives.
After this operation, 133 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Err http://10.30.0.2:8080/ubuntu-part/ trusty/main libasound2-data all 1.0.27.2-3ubuntu7
  404 Not Found
Err http://10.30.0.2:8080/ubuntu-part/ trusty/main libasound2 amd64 1.0.27.2-3ubuntu7
  404 Not Found
Err http://10.30.0.2:8080/ubuntu-part/ trusty/main libasyncns0 amd64 0.8-4ubuntu2
  404 Not Found
Err http://10.30.0.2:8080/ubuntu-part/ trusty/main libatk1.0-data all 2.10.0-2ubuntu2
  404 Not Found
...

Also we have the following errors in /var/log/mirror-sync/ubuntu-update.log log file
[root@fuel mirror-sync]# grep InRelease *
Fri Oct 23 22:53:25 UTC 2015 Fetching '/dists/mos6.1-updates/InRelease' to '/var/www/nailgun/mos-ubuntu/dists/mos6.1-updates/' with params ''...rsync: link_stat "/mos/ubuntu/dists/mos6.1-updates/InRelease" (in mirror) failed: No such file or directory (2)
Fri Oct 23 22:53:36 UTC 2015 Fetching '/dists/mos6.1-security/InRelease' to '/var/www/nailgun/mos-ubuntu/dists/mos6.1-security/' with params ''...rsync: link_stat "/mos/ubuntu/dists/mos6.1-security/InRelease" (in mirror) failed: No such file or directory (2)
Fri Oct 23 22:53:45 UTC 2015 Fetching '/dists/mos6.1-holdback/InRelease' to '/var/www/nailgun/mos-ubuntu/dists/mos6.1-holdback/' with params ''...rsync: link_stat "/mos/ubuntu/dists/mos6.1-holdback/InRelease" (in mirror) failed: No such file or directory (2)
Fri Oct 23 22:54:04 UTC 2015 Fetching '/dists/trusty/InRelease' to '/var/www/nailgun/ubuntu-part/dists/trusty/' with params ''...rsync: link_stat "/dists/trusty/InRelease" (in ubuntu) failed: No such file or directory (2)

[Dmitry Pyzhov (dpyzhov)]

We are working on new version of fuel-createmirror and this issue will not affect it. New version of fuel-createmirror can be used for old environments. So there is no point in fixing this bug.

[Roman Rufanov (rrufanov)]

Customer found on MOS 6.1 - need fix

[Denis Meltsaykin (dmeltsaykin)]

Closing this as Won't Fix per comment from Dmitry Pyzhov.
Roman, please wait for the new version of fuel-createmirror.

[Vitaly Sedelnik (vsedelnik)]

Reopening this bug to check whether fuel-createmirror from 8.0 resolves this. If it does - we need to provide instruction how to update it on existing 6.1 and 7.0 environments.

[Rodion Tikunov (rtikunov)]

fuel-mirror package from 8.0 requires python-fuelclient >= 7.0.0 and 3 packages which not presented in Centos 6:
python-bintrees-2.0.2
python-debian-0.1.23
python-packetary-8.0.0

So, for backporting fuel-mirror to 7.0 we need to build those packages as well as fuel-mirror package.
For backporting to 6.1 we need update python-fuelclient too. But this update may be risky.

[Rodion Tikunov (rtikunov)]

Workaround:
before run fuel-createmirror with option PARTIAL_UPSTREAM=0, delete repos created after the previous run with PARTIAL_UPSTREAM=1
rm -rf /var/www/nailgun/ubuntu-part
rm -rf /var/www/nailgun/mos-ubuntu

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Rodion Tikunov <email address hidden> on branch: stable/7.0
Review: https://review.fuel-infra.org/20505
Reason: Wrong repo

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/314980

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 6.1
Change author: Rodion Tikunov <email address hidden>
Review: https://review.fuel-infra.org/20530

[Rodion Tikunov (rtikunov)]

In 8.0 fuel-createmirror was fully rewritten and renamed to fuel-mirror.
# fuel --version
8.0.0
# fuel-createmirror --help
This script is DEPRECATED. Please use fuel-mirror utility!
Usage: fuel-createmirror [options]
And fuel-mirror has not optioned to create the full upstream Ubuntu mirror.

Anyway you can create full mirror of upstream Ubuntu by using their documentation https://wiki.ubuntu.com/Mirrors

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20530
Submitter: Vitaly Sedelnik <email address hidden>
Branch: 6.1

Commit: b2788c8d8a8983d3a33820ef8f7f24d47ddc90b3
Author: Rodion Tikunov <email address hidden>
Date: Wed May 11 16:30:40 2016

Delete old Packages files before fetching

Partial mirroring builds Packages, Packages[gz, bz2] files for
partial repo. Those files cause problems if after partial mirroring
start full mirror creation.
This patch adds deleting Packages, Packages[gz, bz2] before fetching
those from mirrors.

Closes-Bug: #1510274
Change-Id: Id03afa3b27ab05fac0bbbb5c9b35ba9895735858

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/314980
Committed: https://git.openstack.org/cgit/openstack/fuel-mirror/commit/?id=26aeef185eed79eb2d557198f1c087781f3b3018
Submitter: Jenkins
Branch: stable/7.0

commit 26aeef185eed79eb2d557198f1c087781f3b3018
Author: Rodion Tikunov <email address hidden>
Date: Wed May 11 14:23:34 2016 +0300

    Delete old Packages files before fetching

    Partial mirroring builds Packages, Packages[gz, bz2] files for
    partial repo. Those files cause problems if after partial mirroring
    start full mirror creation.
    This patch adds deleting Packages, Packages[gz, bz2] before fetching
    those from mirrors.

    Change-Id: Ic40a25c0aeae5e899c42553b1bc63ca97f19379d
    Closes-Bug: #1510274

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 6.1 + MU7 updates using steps from description.

Actual result:
Packages collectd, vim-nox are installed successfully.
Output is in https://paste.mirantis.net/show/2472/

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 7.0 + MU5 updates using steps from description.

Actual result:
Packages collectd, vim-nox are installed successfully.
Output is in https://paste.mirantis.net/show/2535/