fuel-agent installs Ubuntu package instead of MOS ones

[Anastasia Palkina]

1. Create new environment
2. Choose Neutron, VLAN
3. Choose Ceph for volumes
4. Choose Murano
5. Add 3 controller, 1 compute, 3 ceph
6. Start deployment
7. Stop deployment during provisioning
8. Wait until nodes become 'Pending addition'
9. Deploy the environment again. It was successful
10. But controller (node-3) has gone away and back online many times

The default set of APT repositories is used. So Ubuntu package installed instead special package for ohai.

root@node-3:~# apt-cache policy ohai
ohai:
  Installed: 6.14.0-2
  Candidate: 6.14.0-2~u14.04+mos1
  Version table:
 *** 6.14.0-2 0
        500 http://archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status
     6.14.0-2~u14.04+mos1 0
       1050 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages
root@node-3:~#

"build_id": "2015-08-25_03-34-40", "build_number": "228", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-25_03-34-40", "build_number": "228", "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "1d23fa825a73564bb6c715993ea9df8a66249a93", "nailgun_sha": "9f227f2cc0ca276715cb098c73b2bd231b103eae", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "e01693992d7a0304d926b922b43f3b747c35964c", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "4c2ab9d6c623d345086c6e2874d1df81fd96a942", "production": "docker", "python-fuelclient_sha": "fc7b63aa6900fe3b2c183108ba6a13e868bc0472", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "3ad03d076c46347691cc3480dd19d34e37b73df4", "release": "7.0", "fuelmain_sha": "28d4bfcff7a0fb1b37504dbcac4998789df17935"

[Alexei Sheplyakov (asheplyakov)]

Apparently fuel-agent installs a wrong version of ohai into the IBP image:

[root@fuel ~]# zcat /var/www/nailgun/targetimages/env_1_ubuntu_1404_amd64.img.gz > /tmp/env1_root.ext4
[root@fuel ~]# mount -t ext4 -o loop,rw,data=ordered,errors=remount-ro /tmp/env1_root.ext4 /mnt
[root@fuel ~]# chroot /mnt apt-cache policy ohai
ohai:
  Installed: 6.14.0-2
  Candidate: 6.14.0-2
  Version table:
 *** 6.14.0-2 0
        100 /var/lib/dpkg/status

[Alexei Sheplyakov (asheplyakov)]

It's fuel-agent bug => reassigning to fuel-python team

[Alexei Sheplyakov (asheplyakov)]

Apparently fuel-agent creates bogus pinning rules which does not match packages from the 'universe' section:

# cat /etc/apt/preferences.d/fuel-image-mos.pref
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=main
Pin-Priority: 1050
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=restricted
Pin-Priority: 1050

These rules does not match packages residing in the `universe' section (which
is where the `ohai' package shipped with Ubuntu belongs to). As a result the original
version of `ohai' package gets installed (which is wrong and re-introduces LP #1463835):

# apt-cache policy ohai
ohai:
  Installed: (none)
  Candidate: 6.14.0-2
  Version table:
     6.14.0-2 0
        500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
     6.14.0-2~u14.04+mos1 0
        500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

Adjusting the pinning rules fixes the problem:

# cat > /etc/apt/preferences.d/fuel-image-mos.pref <<-EOF
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0
Pin-Priority: 1050
EOF

# apt-cache policy ohai
ohai:
  Installed: (none)
  Candidate: 6.14.0-2~u14.04+mos1
  Version table:
     6.14.0-2 0
        500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
     6.14.0-2~u14.04+mos1 0
       1050 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/217108

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/217112

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217112
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=3d18e9e2519c22a9a6404580a64c59c1d86faf21
Submitter: Jenkins
Branch: master

commit 3d18e9e2519c22a9a6404580a64c59c1d86faf21
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Aug 26 15:28:07 2015 +0300

    Fix pinning rules so a package from main can override the one from
    universe

    Current pinning rules don't match packages from the `universe' section.
    As a result a customized MOS package (which always resides in the `main'
    section) has lower priority than its Ubuntu counterpart:

     # apt-cache policy ohai
     ohai:
       Installed: (none)
       Candidate: 6.14.0-2
       Version table:
          6.14.0-2 0
             500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64
    Packages
          6.14.0-2~u14.04+mos1 0
             500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/
    mos7.0/main amd64 Packages

    As a result the original version of package (shipped with Ubuntu) gets
    installed into the IBP image which might reintroduce bugs fixed by
    patched packages and/or break features which require customized
    packages.

    Change-Id: I9326849751a8616fcdd4e3a57297bbb0f49def50
    Co-Authored-By: Alexey Sheplyakov <email address hidden>
    Closes-Bug: #1488844

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217108
Committed: https://git.openstack.org/cgit/stackforge/fuel-agent/commit/?id=bf324e8f3981b050e89f710ab64411ef0d60ff04
Submitter: Jenkins
Branch: master

commit bf324e8f3981b050e89f710ab64411ef0d60ff04
Author: Alexey Sheplyakov <email address hidden>
Date: Wed Aug 26 14:48:23 2015 +0300

    Fix pinning rules so a package from main can override the one from universe

    Current pinning rules don't match packages from the `universe' section.
    As a result a customized MOS package (which always resides in the `main'
    section) has lower priority than its Ubuntu counterpart:

     # apt-cache policy ohai
     ohai:
       Installed: (none)
       Candidate: 6.14.0-2
       Version table:
          6.14.0-2 0
             500 http://10.20.0.1/mirror/ubuntu/ trusty/universe amd64 Packages
          6.14.0-2~u14.04+mos1 0
             500 http://10.20.0.2:8080/2015.1.0-7.0/ubuntu/x86_64/ mos7.0/main amd64 Packages

    As a result the original version of package (shipped with Ubuntu) gets
    installed into the IBP image which might reintroduce bugs fixed by
    patched packages and/or break features which require customized
    packages.

    Closes-Bug: #1488844
    Change-Id: I655b34bf0de80d26ba3f8c25fbb6b17bbaefc076

[Maksym Strukov (unbelll)]

Verified as fixed in 7.0-246

{"build_id": "2015-08-27_18-45-15", "build_number": "246", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "2015-08-27_18-45-15", "build_number": "246", "api": "1.0", "fuel-library_sha": "800b28137e131334d447c353db213b1d43f9bedb", "nailgun_sha": "cbe7dcab16fbdacaa8e80e36a925191e7986a068", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "83048d68609854324ceeaf04242e68d658cfb55d", "release": "7.0", "fuelmain_sha": "8ead1228c637ba80e7dead37dfe09d85f00b7c29"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "800b28137e131334d447c353db213b1d43f9bedb", "nailgun_sha": "cbe7dcab16fbdacaa8e80e36a925191e7986a068", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "8844dc5964fb197b64525f04b6ec15cbbcfe44f0", "production": "docker", "python-fuelclient_sha": "9643fa07f1290071511066804f962f62fe27b512", "astute_sha": "53c86cba593ddbac776ce5a3360240274c20738c", "fuel-ostf_sha": "83048d68609854324ceeaf04242e68d658cfb55d", "release": "7.0", "fuelmain_sha": "8ead1228c637ba80e7dead37dfe09d85f00b7c29"}

[Alexander Gordeev (a-gordeev)]

Hello MOS-Maintenance team, please take a look at this critical bug. It looks like 6.1 is affected too.

[Vladimir Kozhukalov (kozhukalov)]

One can workaround the issue following the steps:
1) Once a node is provisioned, ssh on a node
2) edit /etc/apt/preferences.d/* files so they do not contain section based prefereces
I.e. instead of having many section based stanzas
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0, c=main
Pin-Priority: 1050
it should contain single stanza w/o section
Package: *
Pin: release o=Mirantis, a=mos7.0, n=mos7.0, l=mos7.0
Pin-Priority: 1050
3) run apt-get dist-upgrade

[Polina Petriuk (ppetriuk)]

Would the workaround for MOS6.1 look similar?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/242526

[Sergii Rizvan (srizvan)]

Bug was not reproduced on MOS 6.1. Close as invalid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergii Rizvan (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/242526
Reason: Bug was not reproduced on MOS 6.1.