14.04 package provision didn't succeed on 2.10~27 build

[Vivek]

Server manager is 12.04 27 version. Reimage happens but provisioning gets stuck

root@Blaster-vJCE-063:/tmp/vg# server-manager status server
{
    "server": [
        {
            "id": "cmbu-toystory-01",
            "ip_address": "10.87.129.224",
            "mac_address": "00:25:90:92:0D:EE",
            "status": "compute_started"
        },
        {
            "id": "cmbu-toystory-02",
            "ip_address": "10.87.129.226",
            "mac_address": "00:25:90:92:0E:6E",
            "status": "restart_issued"
        },
        {
            "id": "cmbu-gravity-05",
            "ip_address": "10.87.140.213",
            "mac_address": "00:25:90:AB:9C:78",
            "status": "database_completed"
        }
    ]
}

[Vinod Nair (vinodnair)]

Issue exists in Buid28 also . Storage provisioning does not complete on 14.04

[Michael Nishimoto (mnishimoto)]

Provision Wil Loop until it finishes completely , For the storage failures in 14.04 ,, we need to wait for Dheeraj

following packages have unmet dependencies:
 contrail-storage : Depends: cinder-volume but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Error: /Stage[storage]/Contrail::Storage/Contrail::Lib::Storage_common[storage-compute]/Package[contrail-storage]/ensure: change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install contrail-storage' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 contrail-storage : Depends: cinder-volume but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Thanks
Vinod

From: Vivek Garg
Sent: Tuesday, February 17, 2015 3:36 PM
To: Jeba Paulaiyan; Vinod Nair; Michael Nishimoto; Nitish Krishna Kaveri Poompatnam
Cc: Abhay Joshi; Dheeraj Gautam; Jeya Ganesh Babu Jegatheesan; Wenqing Liang
Subject: RE: Sm provisioning failing in Build 32 also

This is for build 33. 1...

[Dheeraj Gautam (dgautam)]

debugging done so far"

Contrail-storage depends on cinder-volume (no specific version) and
cinder-volume depends on cinder-common. Since latest version of
cinder-volume is chosen for installation, it wants cinder-common to be
latest, but here it find "1:2014.1.3-0ubuntu1~cloud0” version already
installed and fails.

root@cmbu-toystory-01:~# aptitude versions cinder-common
Package cinder-common:
p 1:2014.1-0ubuntu1 trusty 500
i 1:2014.1.3-0ubuntu1~cloud0 <NULL> 999
p 1:2014.1.3-0ubuntu1.1 trusty-security,trusty-updates 500

root@cmbu-toystory-01:~# dpkg --compare-versions
1:2014.1.3-0ubuntu1~cloud0 gt 1:2014.1.3-0ubuntu1.1
root@cmbu-toystory-01:~# echo $?
1
root@cmbu-toystory-01:~# dpkg --compare-versions
1:2014.1.3-0ubuntu1~cloud0 lt 1:2014.1.3-0ubuntu1.1
root@cmbu-toystory-01:~# echo $?
0

Its solution should be add specific version dependency of contrail-storage
and cinder-volume instead of latest version.

[Dheeraj Gautam (dgautam)]

Logs from 14.04 machine (installation via SM)

Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install contrail-storage' returned 100: Reading package lists...
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: Building dependency tree...
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: Reading state information...
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: Some packages could not be installed. This may mean that you have
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: requested an impossible situation or if you are using the unstable
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: distribution that some required packages have not yet been created
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: or been moved out of Incoming.
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: The following information may help to resolve the situation:
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]:
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: The following packages have unmet dependencies:
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: contrail-storage : Depends: cinder-volume but it is not going to be installed
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: E: Unable to correct problems, you have held broken packages.
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: (/Stage[storage]/Contrail::Storage/Contrail::Lib::Storage_common[storage-compute]/Package[contrail-storage]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install contrail-storage' returned 100: Reading package lists...
Feb 10 22:24:09 cmbu-toystory-01 puppet-agent[21358]: (/Stage[storage]/Contrail::Storage/Contrail::Lib::Storage_common[storage-compute]/Package[contrail-storage]/ensure) Building dependency tree…

[Dheeraj Gautam (dgautam)]

some more debugging:

FAB based installs doesn't have access to internet/ubuntu archives. It installs all packages from local repo only. But with puppet based install/re-imaged, nodes have entries for "deb http://us.archive.ubuntu.com/ubuntu/" and this causes to get latest packages to be installed from internet.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7655
Committed: http://github.org/Juniper/contrail-server-manager/commit/6b4e90fd4d3adb2216408c6ce058a671296bc414
Submitter: Zuul
Branch: master

commit 6b4e90fd4d3adb2216408c6ce058a671296bc414
Author: nitishkrishna <email address hidden>
Date: Fri Feb 20 06:34:46 2015 -0800

Closes-Bug: #1420927 - Changing sources.list to minimum required lines (No internet repos)

This check-in:
- Reduces the sources.list on target nodes to few lines
- These lines are: thirdparty packages repo and contrail packages repo on SM
- Local repo with same packages as contrail repo
- Tested on Vinod's setup - both 12.04.3 and 14.04.1

Change-Id: I4c7f76f0e29edb7e470956209e82e6614fdb76f4

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/7654
Committed: http://github.org/Juniper/contrail-server-manager/commit/2766cf5dbc1714c205fbb432c9412a4ff1ba7d41
Submitter: Zuul
Branch: R2.1

commit 2766cf5dbc1714c205fbb432c9412a4ff1ba7d41
Author: nitishkrishna <email address hidden>
Date: Fri Feb 20 06:34:46 2015 -0800

Closes-Bug: #1420927 - Changing sources.list to minimum required lines (No internet repos)

This check-in:
- Reduces the sources.list on target nodes to few lines
- These lines are: thirdparty packages repo and contrail packages repo on SM
- Local repo with same packages as contrail repo
- Tested on Vinod's setup - both 12.04.3 and 14.04.1

Change-Id: I4c7f76f0e29edb7e470956209e82e6614fdb76f4