base system installation fails due to broken deb packages' file names

[Sergey Kolekonov]

The following builds:
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/102/
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/103/

are broken due to provisioning timeout.

I've reverted the second environment and connected to a node using VNC.
The problem is in missing packages. The screenshot is in attachment
I've also found the following errors on nodes (10.108.10.2/var/log/docker-logs/remote/node-1.test.domain.local/anna.log):

2015-04-07T18:17:02.804744+01:00 info: http://10.108.10.2:8080/ubuntu/fuelweb/x86_64/dists/precise-updates/Release:
2015-04-07T18:17:02.805236+01:00 info: 2015-04-07 17:17:03 ERROR 404: Not Found.
2015-04-07T18:17:02.822227+01:00 info: http:///ubuntu/dists/precise-security/Release: Invalid host name.
2015-04-07T18:17:02.854193+01:00 debug: DEBUG: resolver (loop-aes-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854193+01:00 debug: DEBUG: resolver (ufsutils-udeb): package doesn't exist (ignored)
2015-04-07T18:17:02.854218+01:00 debug: DEBUG: resolver (ufs-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854500+01:00 debug: DEBUG: resolver (loop-aes-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.854500+01:00 debug: DEBUG: resolver (ufsutils-udeb): package doesn't exist (ignored)
2015-04-07T18:17:02.854557+01:00 debug: DEBUG: resolver (ufs-modules): package doesn't exist (ignored)
2015-04-07T18:17:02.855473+01:00 debug: DEBUG: retrieving apt-mirror-setup 1:0.55ubuntu4

It can be a mirror problem

[Ivan Berezovskiy (iberezovskiy)]

This issue affects only 5.1.2 staging jobs

[Albert Syriy (asyriy)]

The cause of the issue is lack of the following udeb packages in ISO:

bsdutils 1:2.20.1-1ubuntu3
busybox-initramfs,
diffutils
libusb
dmsetup,
iputils-ping,
libattr1,
libdermapper1.01.1,
libgcc1,
ntpdate,
passwd,
procps,
vim-common,
vim-tiny,
zlib1g

Could we check if if the the packages are on the mirror using for building ISO for 5.1.2 ?

[Albert Syriy (asyriy)]

This is the cause of the Ubuntu provisioning fail.

[Albert Syriy (asyriy)]

All the "missed" packages has the strange substitution in the package name, when the character ':' substituted by the '%3a'.
For example
----Start quotation----
Download complete and in download only mode
I: Calculating obsolete packages
.... (skipped) ....
 -> Processing conffiles for bash
I: Extracting bsdutils_1%3a2.20.1-1ubuntu3.1_amd64.deb...
I: Extracting busybox-initramfs_1%3a1.18.5-1ubuntu4.1_amd64.deb...
... (skipped) ...
---------End the quotation-----
Actually should be ':'
I: Extracting bsdutils_1:2.20.1-1ubuntu3.1_amd64.deb...
I: Extracting busybox-initramfs_1:1.18.5-1ubuntu4.1_amd64.deb...

See the console log for details
http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.iso/121/consoleText

[Albert Syriy (asyriy)]

The status update:

The root cause of the problem is character ':' in file name of packages.
The debootstrap can't process such file name during installation and report about the packages as missing.
Looks like we have a package on our mirrors
bsdutils_1%3a2.20.1-1ubuntu3.1_amd64.deb
Which after processing (building ISO) it is saved as
bsdutils_1:2.20.1-1ubuntu3.1_amd64.deb
and this file name saved in the Packages (file describing the packages in the repository).

After fixing this issue with ":" in names (removing from the file name '1:' characters) met another issue, what investigating now.
( see as an example how the packages kept in Ubuntu repo
http://ru.archive.ubuntu.com/ubuntu/pool/main/u/util-linux/ )

[Alexei Sheplyakov (asheplyakov)]

Actually there are two unrelated bugs:

- mirror/ubuntu/files/mkrepo.sh does not download the base packages properly, instead it copies them
  from multistrap cache. However multistrap modifies the packages' file names (that is, adds the colon
  character and epoch number) which confuses debootstrap. As a result base system installation fails
  (so the provisioning times out).
  The following patch https://review.openstack.org/176309 solves this problem.

- fuel-main builds an ISO with Linux 3.13 [0], however fuel-library insists on installing Linux 3.11 [1].
  To address this issue https://review.openstack.org/152657 should be merged.

The ISO with the above mentioned patches has successfully passed BVTs [2]

[0] https://github.com/stackforge/fuel-main/blob/stable/5.1/config.mk#L103

[1] https://github.com/stackforge/fuel-library/blob/stable/5.1/deployment/puppet/cobbler/templates/preseed/ubuntu-1204.preseed.erb#L150

[2] http://jenkins-product.srt.mirantis.net:8080/job/5.1.2.staging.ubuntu.bvt_2/120

[ruhe (ruhe)]

Cleaned up the bug a little bit. Sorry for spam if there was any.

[Alexei Sheplyakov (asheplyakov)]

Fixed by https://review.openstack.org/176309 and https://review.openstack.org/152657