Can't build iso from fuel-main

[Jouston Huang]

Build OS: Ubuntu 12.04.3 LTS

Follow http://docs.mirantis.com/fuel-dev/develop/env.html to setup build environment
git clone https://github.com/stackforge/fuel-main
make iso USE_MIRROR=ext

The system failed to make an iso

http://pastebin.com/RetBCAhy

http://osci-obs.vm.mirantis.net:82/centos-fuel-4.0-stable/centos/repodata/repomd.xml: [Errno 14] PYCURL ERROR 6 - ""
Trying other mirror.
Cannot retrieve repository metadata (repomd.xml) for repository: fuel. Please verify its path and try again
# Yumdownloader/repotrack workaround number one:
# i686 packages are downloaded by mistake. Remove them
rm -rf /home/jouston/Project/fuel-main/local_mirror/centos/os/x86_64/Packages/*.i686.rpm
# Yumdownloader workaround number two:
# yumdownloader should fail if some packages are missed
test `grep "No Match" /home/jouston/Project/fuel-main/build/mirror/centos/yumdownloader.log | wc -l` = 0
# Yumdownloader workaround number three:
# We have exactly two downloading conflicts: django and mysql
test `grep "has depsolving problems" /home/jouston/Project/fuel-main/build/mirror/centos/yumdownloader.log | wc -l` = 2
make: *** [/home/jouston/Project/fuel-main/build/mirror/centos/yum.done] Error 1

[Sergey Vasilenko (xenolog)]

I think should be USE_MIRROR=ext YUM_REPOS=official

[Jouston Huang (jouston-huang)]

@xenolog: I have checked config.mk. YUM_REPOS will be suppressed by "USE_MIRROR=ext". Seems like the http://download.mirantis.com/fuelweb-repo/3.2/ nor http://download.mirantis.com/fuelweb-repo/4.0 isn't complete.

The error message is

Cannot retrieve repository metadata (repomd.xml) for repository: fuel. Please verify its path and try again

[Jouston Huang (jouston-huang)]

check ~fuel-main/build/mirror/centos/yumdownloader.log

There are some errors:
Could not download/verify pkg lzop-1.02-0.9.rc1.el6.x86_64: failure: Packages/lzop-1.02-0.9.rc1.el6.x86_64.rpm from proprietary: [Errno 256] No more mirrors to try.
http://download.mirantis.com/fuelweb-repo/3.2/centos/os/x86_64/Packages/mailcap-2.1.31-2.el6.noarch.rpm: [Errno 14] HTTP Error 416 : http://download.mirantis.com/fuelweb-repo/3.2/centos/os/x86_64/Packages/mailcap-2.1.31-2.el6.noarch.rpm

After manually wget all packages I get pass to the next step.

make clean; make iso USE_MIRROR=ext MIRROR_CENTOS?=http://download.mirantis.com/fuelweb-repo/3.2/centos MIRROR_UBUNTU?=http://archive.ubuntu.com/ubuntu

Now I saw these:

W: Failed to fetch http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-3.2-stable/reprepro/dists/precise/InRelease

W: Failed to fetch http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-3.2-stable/reprepro/dists/precise/Release.gpg Could not resolve 'osci-obs.vm.mirantis.net'

W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
  dbus libaccountsservice0 libdbus-glib-1-2 libpolkit-gobject-1-0
Suggested packages:
  gnome-control-center dbus-x11
The following NEW packages will be installed:
  accountsservice dbus libaccountsservice0 libdbus-glib-1-2 libpolkit-gobject-1-0
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/555 kB of archives.
After this operation, 1810 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libdbus-glib-1-2 libpolkit-gobject-1-0 libaccountsservice0 dbus accountsservice
Authentication warning overridden.
Download complete and in download only mode
Reading package lists... Done
Building dependency tree... Done
The following NEW packages will be installed:
  acl
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/42.9 kB of archives.
After this operation, 188 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  acl
Authentication warning overridden.
Download complete and in download only mode
Reading package lists... Done
Building dependency tree... Done
adduser is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package alembic
make: *** [/home/jouston/Project/fuel-main/build/mirror/ubuntu/createchroot.done] Error 1

[Dmitry Pyzhov (dpyzhov)]

Are you building current master iso or 3.2-release?

[Jouston Huang (jouston-huang)]

@lux-place: I'm building tag:3.2

BTW. More findings here. Now my build script can make it a little bit further.

" make clean; make iso USE_MIRROR=ext MIRROR_CENTOS?=http://download.mirantis.com/fuelweb-repo/3.2/centos MIRROR_UBUNTU?=http://archive.ubuntu.com/ubuntu MIRROR_FUEL_UBUNTU?=http://download.mirantis.com/fuelweb-repo/4.0/ubuntu/ "

And the make process have been gone through

$(BUILD_DIR)/mirror/ubuntu/createchroot.done
 (DONE)mkdir -p $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot
 (DONE)sudo debootstrap --no-check-gpg --include=wget --components=main,universe,multiverse,restricted $(UBUNTU_RELEASE) $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot $(MIRROR_UBUNTU)
 (DONE)echo deb $(MIRROR_FUEL_UBUNTU) $(UBUNTU_RELEASE) main | sudo tee $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/etc/apt/sources.list.d/mirantis.list
 (DONE)echo 'APT::Get::AllowUnauthenticated 1;' | sudo tee $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/etc/apt/apt.conf.d/02mirantis-unauthenticated
 (DONE)sudo cp -a $(SOURCE_DIR)/mirror/ubuntu/files/preferences $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/etc/apt
 (DONE)sudo cp -a $(SOURCE_DIR)/requirements-deb.txt $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/
 (DONE)sudo mkdir -p $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/repo
 (DONE)sudo rsync -a $(SOURCE_DIR)/mirror/ubuntu/files/ $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot/repo/
 sudo chroot $(LOCAL_MIRROR_UBUNTU_OS_BASEURL)/chroot /bin/bash -c "chmod +x /repo/mkrepo.sh && /repo/mkrepo.sh" <--- Failed here!!!

Error message:
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package alembic
make: *** [/home/jouston/Project/fuel-main/build/mirror/ubuntu/createchroot.done] Error 1

I've chroot in and execute /repo/mkrepo.sh . Unfortunately the apt-get couldn't find alembic at all.

[Jouston Huang (jouston-huang)]

It's a weired bug. From yesterday I change my build script and going further.

make deep_clean; make iso USE_MIRROR=ext MIRROR_UBUNTU?=http://tw.archive.ubuntu.com/ubuntu

touch /home/jouston/Project/fuel-main/build/repos/fuellib.done
mkdir -p /home/jouston/Project/fuel-main/build/mirror/eggs
sudo sh -c "${SANDBOX_UP}"
rpm: RPM should not be used directly install RPM packages, use Alien instead!
rpm: However assuming you know what you are doing...
warning: /home/jouston/Project/fuel-main/local_mirror/centos/os/x86_64/Packages/centos-release-6-4.el6.centos.10.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
 package centos-release-6-4.el6.centos.10.x86_64 is intended for a different architecture
centos-release already installed
Config time: 0.024
Yum Version: 3.2.25
Setting up Package Sacks
mirror | 1.1 kB 00:00 ...
mirror/primary | 606 kB 00:00 ...
mirror 1132/1132
pkgsack time: 3.079
rpmdb time: 0.000
Setting up Install Process
Checking for virtual provide or file-provide for byacc
Searching pkgSack for dep: byacc
No package byacc available.
Checking for virtual provide or file-provide for flex
Searching pkgSack for dep: flex
No package flex available.
Checking for virtual provide or file-provide for gcc
Searching pkgSack for dep: gcc
No package gcc available.
Checking for virtual provide or file-provide for glibc-devel
Searching pkgSack for dep: glibc-devel
No package glibc-devel available.
Checking for virtual provide or file-provide for glibc-headers
Searching pkgSack for dep: glibc-headers
No package glibc-headers available.
Checking for virtual provide or file-provide for kernel-headers
Searching pkgSack for dep: kernel-headers
No package kernel-headers available.
Checking for virtual provide or file-provide for make
Searching pkgSack for dep: make
No package make available.
Checking for virtual provide or file-provide for openssl-devel
Searching pkgSack for dep: openssl-devel
No package openssl-devel available.
Checking for virtual provide or file-provide for postgresql-devel
Searching pkgSack for dep: postgresql-devel
No package postgresql-devel available.
Checking for virtual provide or file-provide for python-devel.x86_64
Searching pkgSack for dep: python-devel.x86_64
No package python-devel.x86_64 available.
Building updates object
up:Obs Init time: 0.004
up:simple updates time: 0.001
up:obs time: 0.000
up:condense time: 0.000
updates time: 0.022
Checking for virtual provide or file-provide for rpm-build
Searching pkgSack for dep: rpm-build
No package rpm-build available.
Checking for virtual provide or file-provide for ruby
Searching pkgSack for dep: ruby
No package ruby available.
Checking for virtual provide or file-provide for ruby-devel
Searching pkgSack for dep: ruby-devel
No package ruby-devel available.
Checking for virtual provide or file-provide for tar
Searching pkgSack for dep: t...

[Roman Vyalov (r0mikiam)]

please run:
make deep_clean; make iso USE_MIRROR=ext MIRROR_UBUNTU?=http://tw.archive.ubuntu.com/ubuntu FUELMAIN_COMMIT=3.2-fixes FUELLIB_COMMIT=3.2-fixes NAILGUN_COMMIT=3.2-fixes ASTUTE_COMMIT=3.2-fixes OSTF_COMMIT=3.2-fixes

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59366
Committed: http://github.com/stackforge/fuel-main/commit/817d9ba7c41b24d1c8c3ed882ec64b1fd75eda44
Submitter: Jenkins
Branch: 3.2-fixes

commit 817d9ba7c41b24d1c8c3ed882ec64b1fd75eda44
Author: Roman <email address hidden>
Date: Mon Dec 2 17:31:29 2013 +0400

    change default branch to 3.2-fixes for all fuel component

    Change-Id: I2c596c72550abab6d8d2a6fe2ceecd834ae2fb46
    Closes-Bug: #1255088

[Jouston Huang (jouston-huang)]

@r0mikiam I have tried to run:

make deep_clean; make iso USE_MIRROR=ext MIRROR_UBUNTU?=http://tw.archive.ubuntu.com/ubuntu FUELMAIN_COMMIT=3.2-fixes FUELLIB_COMMIT=3.2-fixes NAILGUN_COMMIT=3.2-fixes ASTUTE_COMMIT=3.2-fixes OSTF_COMMIT=3.2-fixes

Unfortunately it's running into django check.

So I modified mirror/centos/repo.mk @line.51 (see attachment for patch)
mdownloader.log | wc -l` = 2

We also need to change the build parameter to:

make deep_clean ; make iso USE_MIRROR=ext YUM_REPOS=official FUELMAIN_COMMIT=3.2-fixes FUELLIB_COMMIT=3.2-fixes NAILGUN_COMMIT=3.2-fixes ASTUTE_COMMIT=3.2-fixes OSTF_COMMIT=3.2-fixes

The reason why I take "MIRROR_UBUNTU?=http://tw.archive.ubuntu.com/ubuntu" off because when creating chroot it can’t find “alembic” package.

Then I run into this issue:
touch /home/jouston/Project/fuel-main/build/mirror/rhel/build.done
# Clone repo and checkout required commit
mkdir -p /home/jouston/Project/fuel-main/build/repos
rm -rf /home/jouston/Project/fuel-main/build/repos/fuellib
git clone https://github.com/Mirantis/fuel.git /home/jouston/Project/fuel-main/build/repos/fuellib
Cloning into '/home/jouston/Project/fuel-main/build/repos/fuellib'...
remote: Counting objects: 57029, done.
remote: Compressing objects: 100% (23327/23327), done.
remote: Total 57029 (delta 29680), reused 50469 (delta 23599)
Receiving objects: 100% (57029/57029), 34.01 MiB | 3.09 MiB/s, done.
Resolving deltas: 100% (29680/29680), done.
cd /home/jouston/Project/fuel-main/build/repos/fuellib && git reset --hard 3.2-fixes
fatal: ambiguous argument '3.2-fixes': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
make: *** [/home/jouston/Project/fuel-main/build/repos/fuellib.done] Error 128

[Jouston Huang (jouston-huang)]

Apologized that I've change the build environment to 13.04.

[Roman Vyalov (r0mikiam)]

fix merged to 3.2-fixes branch. please update repository. checkout 3.2-fixes branch and run . make clean; make iso USE_MIRROR=ext

Building iso tested to precise 12.04

[Jouston Huang (jouston-huang)]

Successfully built iso.

system: ubuntu 13.04 64bit on VirtualBox

Steps:
1. git clone https://review.openstack.org/stackforge/fuel-main
2. cd fuel-main ; git checkout 3.2-fixes
3. remark ./mirror/centos/repo.mk line 51.
4. make clean ; make iso
5. successfully make iso file

[Jouston Huang (jouston-huang)]

Tried to reproduce the make but failed again.

system: ubuntu 12.04 LTS 32bit on Bare Metal

Steps:
1. git clone https://review.openstack.org/stackforge/fuel-main
2. cd fuel-main ; git checkout 3.2-fixes
3. remark ./mirror/centos/repo.mk line 51.
4. make clean ; make iso

Seems the ubuntu mirror isn't complete to cause this issue:

sudo debootstrap --no-check-gpg --include=wget --components=main,universe,multiverse,restrict
ed precise /home/jouston/Project/fuel-main/local_mirror/ubuntu/chroot http://download.miranti
s.com/fuelweb-repo/3.2.1/ubuntu
I: Retrieving Release
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://download.mirantis.com/fuelweb-repo/3.2.1/ubuntu...
E: Couldn't find these debs: wget
make: *** [/home/jouston/Project/fuel-main/build/mirror/ubuntu/createchroot.done] Error 1

While using USE_MIRROR=ext it used to be http://download.mirantis.com/fuelweb-repo/3.2/ubuntu but not it's http://download.mirantis.com/fuelweb-repo/3.2.1/ubuntu

[Dmitry Pyzhov (dpyzhov)]

We need to make sure, but it seems that we don't support build on 32bit systems. Documentation will be updated after the check.

[Jouston Huang (jouston-huang)]

Confirmed that only 64bit system was supported and iso can be build by simply checkout 3.2-fixes code and follow the document.

http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso

[Jouston Huang (jouston-huang)]

Fix confirmed. Successfully build iso from 3.2-fixes git code.