Some packages weren't upgraded after cluster update to 9.2MU

[Ilya Bumarskov]

Fuel 9.2 (MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2017-06-20-142429, MOS_UBUNTU_MIRROR_ID=9.0-2017-06-20-142429)

Steps to reproduce:
   - Deploy 9.1 env with following nodes:
         * Controller
         * Controller
         * Controller
         * Compute, ceph-osd
         * Compute, ceph-osd
         * Compute, ceph-osd
   - Update cluster following the instructions: https://docs.mirantis.com/openstack/fuel/fuel-9.2/release-notes/update-product.html

Observed behavior:
Some packages weren't upgraded. For example:
root@node-1:~# apt-cache policy qemu
qemu:
  Installed: 1:2.3+dfsg-5~u14.04+mos3
  Candidate: 1:2.5+dfsg-5ubuntu6~u1404+mos4
  Version table:
     1:2.5+dfsg-5ubuntu6~u1404+mos4 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2017-06-20-142429/ mos9.0-proposed/main amd64 Packages
 *** 1:2.3+dfsg-5~u14.04+mos3 0
       1050 http://mirror.fuel-infra.org/mos-repos/ubuntu/9.0/ mos9.0-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1:2.3+dfsg-5~u14.04+mos2 0
       1050 http://10.109.0.2:8080/mitaka-9.0/ubuntu/x86_64/ mos9.0/main amd64 Packages
     2.0.0+dfsg-2ubuntu1.34 0
        500 http://archive.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
        500 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
     2.0.0~rc1+dfsg-0ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages

[Max Yatsenko (myatsenko)]

    Please find below information about why some packages are not updated
after cluster update to 9.2MU .

The main reason - is that 'get_packages_for_upgrade.rb' puppet function doesn't return necessary list of packages (it returns a hash of packages).

To get needed packages for upgrade 'get_packages_for_upgrade.rb' script runs this shell command:
'apt-get --just-print -o Dir::etc::sourcelist='-' -o Dir::Etc::sourceparts='#{source_list}' dist-upgrade -qq'

, where 'source_list' - is a path to a directory, where repo file is located
 (it's hardcoded to '/etc/fuel/maintenance/apt/sources.list.d/').

In fact '/etc/fuel/maintenance/apt/sources.list.d/' directory contains a symbol link to a repository file located in standard place (/etc/apt/sources.list.d/),
for "mos9.2-updates.list" repo it contains the following:
mos9.2-updates.list -> /etc/apt/sources.list.d/mos9.2-updates.list

After run 'apt-get --just-print -o Dir::etc::sourcelist='-'
-o Dir::Etc::sourceparts='#{source_list}' dist-upgrade -qq' there is output like this:
....
..
Remv nova-compute-qemu [2:13.1.1-7~u14.04+mos20] [nova-compute:amd64 ]
Remv qemu [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-kvm [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-system [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-system-arm [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-system-ppc [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-system-mips [1:2.3+dfsg-5~u14.04+mos3]
Remv qemu-system-common [1:2.3+dfsg-5~u14.04+mos3] [qemu-system-x86:amd64 qemu-system-sparc:amd64 qemu-system-misc:amd64 ]
Remv qemu-system-misc [1:2.3+dfsg-5~u14.04+mos3] [qemu-system-x86:amd64 qemu-system-sparc:amd64 ]
Remv qemu-system-sparc [1:2.3+dfsg-5~u14.04+mos3] [qemu-system-x86:amd64 ]
Remv qemu-system-x86 [1:2.3+dfsg-5~u14.04+mos3]
Inst libxml2 [2.9.1+dfsg1-3ubuntu4.9] (2.9.3+dfsg1-1~u1404+mos2 mos9.0:mos9.0-updates [amd64])
Inst ceph [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst ceph-common [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst python-rbd [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst python-rados [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst librbd1 [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst librados2 [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
Inst python-cephfs [0.94.6-1~u14.04+mos1] (0.94.9-1~u14.04+mos2 mos9.0:mos9.0-updates [amd64]) []
..
.......

'get_packages_for_upgrade.rb' script parses this output and select all lines that contain 'Inst' operation label.
Here is a line of code that performs it:
'package_list = package_list_common_out.split("\n").select { |line| line =~ /^Inst\b/ }'

Afer this parsing all packages with 'Remv' operation label are missed, they aren't returned to puppet manifest for package resource creation, package upgrade performing.
So it leads to a situation that is described in 'Bug Description': not all packages are upgraded after applying 9.2MU.

 Why there is 'Remv' operation label ?
    When we run:
'apt-get --just-print -o Dir::etc::sourcelist='-'
 -o Dir::Etc::sourceparts='/etc/fuel/maintenance/apt/sour...

[Denis Meltsaykin (dmeltsaykin)]

Moving the bug to 9.2-Mu-4 since it's non trivial to solve.