stx-tools: failed to download required packaged, block ceph upgrade verify

[Changcheng Intel]

Title
-----
stx-tools: failed to download required packages, block ceph upgrade verify

Brief Description
-----------------
Follow https://docs.starlingx.io/developer_guide/index.html to setup docker container and run "download_mirror.sh", it always failed to download the required rpms.

Severity
--------
Critical, block verifying ceph upgrade.

Steps to Reproduce
------------------
Run below command in docker centos container with following instructions in https://docs.starlingx.io/developer_guide/index.html
git clone https://git.starlingx.io/stx-tools
cd stx-tools
git checkout -b reproduce dd66d0e957d95f986f
Set the docker container and run below command in the container
# bash download_mirror.sh

Expected Behavior
------------------
No error when downloading the the required files

Logs
-----------------
18-11-21 11_24_27: failure: repodata/2b47456eecd3ea93ae5d00c9b86372c11f1e6f8572a863dedab256639764dd81-primary.sqlite.bz2 from Starlingx-buildlogs.centos.org_c7.1708.u.i386: [Errno 256] No more mirrors to try.
18-11-21 11_24_28: ERROR: Something wrong with downloading files listed in ./rpms_3rdparties.lst.
18-11-21 12_43_39: failure: repodata/2b47456eecd3ea93ae5d00c9b86372c11f1e6f8572a863dedab256639764dd81-primary.sqlite.bz2 from Starlingx-buildlogs.centos.org_c7.1708.u.i386: [Errno 256] No more mirrors to try.
18-11-21 12_43_40: ERROR: Something wrong with downloading files listed in ./rpms_centos3rdparties.lst.

[Changcheng Intel (liuc-intel)]

I'm using below ways to run "download_mirror.sh"

1. get stx-tools
cd $HOME
git clone https://git.starlingx.io/stx-tools
cd stx-tools
git checkout -b erich_base 1d2ecbb0be2

2. apply below patch
nstcc1@nstcloudcc1:stx-tools$ git diff
diff --git a/centos-mirror-tools/Dockerfile b/centos-mirror-tools/Dockerfile
index eb1f457..28569b2 100644
--- a/centos-mirror-tools/Dockerfile
+++ b/centos-mirror-tools/Dockerfile
@@ -4,13 +4,13 @@ FROM centos:7.4.1708

 WORKDIR /localdisk

-#ENV http_proxy "http://your.actual_http_proxy.com:your_port"
-#ENV https_proxy "https://your.actual_https_proxy.com:your_port"
+ENV http_proxy "http://child-prc.intel.com:913"
+ENV https_proxy "https://child-prc.intel.com:913"
 #ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"

-#RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
-# echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
-#export ftp_proxy=$ftp_proxy" >> /root/.bashrc
+RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
+ echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\n\
+export ftp_proxy=$ftp_proxy" >> /root/.bashrc

 RUN yum install -y epel-release sudo vim-enhanced net-tools git \
         /usr/bin/yumdownloader rpm-build rpm-sign deltarpm wget bind \

3. build docker image
cd centos-mirror-tools
docker build --build-arg https_proxy=http://child-prc.intel.com:913 --build-arg http_proxy=http://child-prc.intel.com:913 --tag $USER:centos-mirror-repository -f Dockerfile .

4. run docker image into the container
docker run -it --name $USER-centos-mirror-repository --volume $(pwd):/localdisk $USER:centos-mirror-repository

5. execute below command in the container
bash download_mirror.sh

6. result:
[root@90104f541da1 logs]# wc -l *
      0 K1_failmoved_url_centos.log
      0 K1_rpms_found_centos.log
      0 K1_rpms_missing_centos.log
      0 K1_srpms_found_centos.log
      0 K1_srpms_missing_centos.log
     46 L1_failmoved_url_3rdparties.log
   5306 L1_failmoved_url_centos.log
    394 L1_failmoved_url_centos3rdparties.log
      0 L1_rpms_found_3rdparties.log
      0 L1_rpms_found_centos.log
      0 L1_rpms_found_centos3rdparties.log
      0 L1_rpms_missing_3rdparties.log
      0 L1_rpms_missing_centos.log
      0 L1_rpms_missing_centos3rdparties.log
     23 L1_srpms_found_3rdparties.log
   1936 L1_srpms_found_centos.log
    148 L1_srpms_found_centos3rdparties.log
      0 L1_srpms_missing_3rdparties.log
      0 L1_srpms_missing_centos.log
      0 L1_srpms_missing_centos3rdparties.log
    590 all_i686.txt
   9858 otherfiles_centos_download.log
      1 rpm-gpg-key-missing.txt
     80 rpms_3rdparties_download_L1.log
    490 rpms_centos3rdparties_download_L1.log
      2 rpms_centos_download_K1.log
   6735 rpms_centos_download_L1.log
  57768 tarballs_download.log
  83377 total

I've uploaded all logs into erich_base_v1.tar.bz2

[Changcheng Intel (liuc-intel)]

No good luck to do “build-pkgs” either.

1. get stx-tools
cd $HOME
git clone https://git.starlingx.io/stx-tools
cd stx-tools
git checkout -b erich_base 1d2ecbb0be2

2. apply below patch
nstcc1@nstcloudcc1:stx-tools$ git diff Dockerfile tb.sh Makefile
diff --git a/Dockerfile b/Dockerfile
index 87ef00b..dd730f9 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -13,9 +13,9 @@
 FROM centos:7.4.1708

 # Proxy configuration
-#ENV http_proxy "http://your.actual_http_proxy.com:your_port"
-#ENV https_proxy "https://your.actual_https_proxy.com:your_port"
-#ENV ftp_proxy "http://your.actual_ftp_proxy.com:your_port"
+ENV http_proxy "http://child-prc.intel.com:913"
+ENV https_proxy "http://child-prc.intel.com:913"
+ENV ftp_proxy "ftp://child-prc.intel.com:913"

 # username you will docker exec into the container as.
@@ -26,6 +26,8 @@ ARG MYUID=1000

 ENV container=docker

+RUN echo "proxy=$http_proxy" >> /etc/yum.conf && \
+echo -e "export http_proxy=$http_proxy\nexport https_proxy=$https_proxy\nexport ftp_proxy=$ftp_proxy" >> /root/.bashrc
 # Download required dependencies by mirror/build processes.
 # Notice there are 3 invocations to yum package manage.
 # 1) Enable EPEL repository.
diff --git a/Makefile b/Makefile
index c4ebff4..73f5f3a 100644
--- a/Makefile
+++ b/Makefile
@@ -20,6 +20,8 @@ all:
        docker build \
                --build-arg MYUID=$(UID) \
                --build-arg MYUNAME=$(MYUNAME) \
+ --build-arg https_proxy=http://child-prc.intel.com:913 \
+ --build-arg http_proxy=http://child-prc.intel.com:913 \
                --ulimit core=0 \
                --network host \
                -t $(TC_CONTAINER_TAG) \
diff --git a/tb.sh b/tb.sh
index e0508dd..9530efd 100755
--- a/tb.sh
+++ b/tb.sh
@@ -38,11 +38,14 @@ function run_container {
     docker run -it --rm \
         --name ${TC_CONTAINER_NAME} \
         --detach \
+ -v /etc/localtime:/etc/localtime:ro \
         -v ${LOCALDISK}:/${GUEST_LOCALDISK} \
         -v ${HOST_MIRROR_DIR}:/import/mirrors:ro \
         -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
         -v ~/.ssh:/mySSH:ro \
         -e "container=docker" \
+ --env https_proxy=http://child-prc.intel.com:913 \
+ --env http_proxy=http://child-prc.intel.com:913 \
         -e MYUNAME=${MYUNAME} \
         --privileged=true \
         --security-opt seccomp=unconfined \

3. build docker image
make all

4. run docker image
bash tb.sh run

5. execute into docker container
bash tb.sh exec

6.
[nstcc1@d8ae703a2a91 /]$ eval $(ssh-agent)
Agent pid 15524

[nstcc1@d8ae703a2a91 /]$ ssh-add
Identity added: /home/nstcc1/.ssh/id_rsa (/home/nstcc1/.ssh/id_rsa)
[nstcc1@d8ae703a2a91 /]$ cd $MY_REPO_ROOT_DIR

[nstcc1@d8ae703a2a91 starlingx]$ repo init -u https://git.starlingx.io/stx-manifest -m default.xml

[nstcc1@d8ae703a2a91 starlingx]$ repo sync -j4

[nstcc1@d8ae703a2a91 starlingx]$ ln -s /import/mirrors/CentOS/stx-r1/CentOS/pike/downloads/ $MY_REPO/stx/

[nstcc1@d8ae703a2a91 starlingx]$ populate_downloads.sh /import/mirrors/CentOS/stx-r1/CentOS/pike/

7.
nstcc1@nstcloudcc1:~$ mkdir -p $HOME/starlingx/mirror/CentOS/tis-installer

nstcc1@nstcloudcc1:~$ cp $HOME/starlingx/mirror/CentOS/stx...

[Ghada Khalil (gkhalil)]

Marking as release gating as this impacts the ceph upgrade feature

[Changcheng Intel (liuc-intel)]

This is my running container proxy env:
[nstcc1@d8ae703a2a91 /]$ env | grep -i proxy
NO_PROXY=localhost,.intel.com,127.0.0.0/8,172,16.0.0/20,192.168.0.0/16,10.0.0.0/8
http_proxy=http://child-prc.intel.com:913
FTP_PROXY=ftp://child-prc.intel.com:913
ftp_proxy=ftp://child-prc.intel.com:913
https_proxy=http://child-prc.intel.com:913
HTTPS_PROXY=http://child-prc.intel.com:913
no_proxy=localhost,.intel.com,127.0.0.0/8,172,16.0.0/20,192.168.0.0/16,10.0.0.0/8
HTTP_PROXY=http://child-prc.intel.com:913

[Changcheng Intel (liuc-intel)]

Need do below change in the running container shared by <email address hidden>
     sudo sed -i 's/nr_inodes=0/nr_inodes=100k/g' /usr/lib/python2.7/site-packages/mockbuild/plugins/tmpfs.py

Feedback build result later.

[Changcheng Intel (liuc-intel)]

The above change could make build some other pkgs successfully. However, I still hit error when building ceph.

09:34:03 Mock Version: 1.3.4
09:34:03 INFO: Mock Version: 1.3.4
09:34:03 Finish: chroot init
09:34:03 Start: build phase for ceph-10.2.6-0.el7.tis.3.src.rpm
09:34:03 Start: build setup for ceph-10.2.6-0.el7.tis.3.src.rpm
09:34:03 ERROR: Exception(/localdisk/loadbuild/nstcc1/starlingx/std/rpmbuild/SRPMS/ceph-10.2.6-0.el7.tis.3.src.rpm) Config(mock/b0) 0 minutes 7 seconds
09:34:03 INFO: Results and/or logs in: /localdisk/loadbuild/nstcc1/starlingx/std/results/nstcc1-starlingx-tis-r5-pike-std/ceph-10.2.6-0.el7.tis.3
09:34:03 ERROR: Command failed:
09:34:03 # /usr/bin/yum-builddep --installroot /localdisk/loadbuild/nstcc1/starlingx/std/mock/b0/root/ --releasever 7 /localdisk/loadbuild/nstcc1/starlingx/std/mock/b0/ro
ot//builddir/build/SRPMS/ceph-10.2.6-0.el7.tis.3.src.rpm
09:34:03 Failed to set locale, defaulting to C
09:34:03 http://127.0.0.1:8088/localdisk/loadbuild/nstcc1/starlingx/installer/rpmbuild/RPMS/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found

[Changcheng Intel (liuc-intel)]

Currently, I could build out bootimage.iso when using below default.xml
    nstcc1@nstcloudcc1:manifests$ git remote -v
    origin https://git.starlingx.io/stx-manifest (fetch)
    origin https://git.starlingx.io/stx-manifest (push)
    nstcc1@nstcloudcc1:manifests$ git status
    On branch default
    Your branch is up to date with 'origin/master'.

    nothing to commit, working tree clean
    nstcc1@nstcloudcc1:manifests$ git rev-parse HEAD
    cfc5d2249fc01d2aa06cff91680a9931c21d4a51
    nstcc1@nstcloudcc1:manifests$

[Changcheng Intel (liuc-intel)]

Please close this problem. We'll file new bug if we meet with new build problem when using the new default.xml