fuellib_review_pkgs_master_node does not use packages built as part of change

[Ryan Moe]

This jobs builds new packages but they are not used when deploying environments. See this run [0] for an example.

A new fuel-ha-utils package was built ( fuel-ha-utils_8.0.0-7290.2.gerrit225941.6.gitea9fe4b_all.deb) but in the failed deployment the controller had 8.0.0-7288.1.gitacfcfd2 installed. The package built by the job included the correct changes. I was not able to look at the contents of the installed package but because the test failed I can safely assume they did not include the necessary changes. The job was triggered by this review [1].

[0] https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4775/
[1] https://review.openstack.org/#/c/225941/

[Sergey Kulanov (skulanov)]

Some update, from discussion with Artem:

No aux repo in pinning rules:

root@node-1:~# cat /etc/apt/preferences.d/*
Package: *
Pin: release
Pin-Priority: 1150Package: *
Pin: release o=Mirantis,a=mos8.0-holdback,l=mos8.0,n=mos8.0-holdback
Pin-Priority: 1100Package: *
Pin: release n=mos8.0,o=Mirantis,l=mos8.0,a=mos8.0
Pin-Priority: 1050Package: *
Pin: release o=Mirantis,n=mos8.0-security,a=mos8.0-security,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release a=mos8.0-updates,o=Mirantis,n=mos8.0-updates,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,a=trusty-updates,n=trusty,v=14.04
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty-security

So built packages in aux repo, has default pinning rules:

root@node-1:~# apt-cache policy fuel-misc
fuel-misc:
  Installed: 8.0.0-7288.1.gitacfcfd2
  Candidate: 8.0.0-7288.1.gitacfcfd2
  Version table:
     8.0.0-7292.1.git84cdfdf 0
        500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
 *** 8.0.0-7288.1.gitacfcfd2 0
       1050 http://10.109.5.2:8080/2015.1.0-8.0/ubuntu/x86_64/ mos8.0/main amd64 Packages
        100 /var/lib/dpkg/status

[Sergey Kulanov (skulanov)]

Set to critical, since we have CI impact

[Sergey Kulanov (skulanov)]

Critical for 7.0 since the same issue:

fuel-snapshot-2015-10-09_04-01-08/node-1.test.domain.local/commands/ubuntu_repo_list.txt

[node-1.test.domain.local] out: 1100 http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/ mos7.0-holdback/main amd64 Packages
[node-1.test.domain.local] out: release o=Mirantis,a=mos7.0-holdback,n=mos7.0,l=mos7.0,c=main
[node-1.test.domain.local] out: origin mirror.fuel-infra.org
[node-1.test.domain.local] out: 500 http://10.109.0.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.0.2
[node-1.test.domain.local] out: Pinned packages:
[node-1.test.domain.local] out:

[Maksym Strukov (unbelll)]

Ryan's run [0] was analyzed. In agree with snapshot, all three nodes has correct record for aux repo in astute.yaml:
'''
  - name: Auxiliary
    section: main
    uri: http://10.109.5.2:8080/ubuntu/auxiliary/
    priority: 1150
    suite: auxiliary
    type: deb
'''

But after provisioning in /etc/apt/preferences.d/* aux repo has wrong priority:
'''
500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
release c=main
origin 10.109.5.2
'''

Reassigned to fuel-lib for further investigation, because it's all ok on fuel-qa side.

---
[0] https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4775/artifact/logs/4775/fail_error_deploy_neutron_tun-fuel-snapshot-2015-10-02_14-17-34.tar.xz

[Matthew Mosesohn (raytrac3r)]

On my locally deployed env, I see
root@node-2:~# apt-cache policy | grep -A3 aux
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/restricted amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/main amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2

From that snapshot from the CI job, it's:
[mmosesohn@localhost commands]$ grep -A3 aux ubuntu_repo_list.txt
[node-1.test.domain.local] out: 500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.5.2

The regenerate-ubuntu-repo script is generating the wrong Release metadata.
currently it has :
release c=main

but it needs to have:
release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=main

The pinning file depends on generating the correct Release metadata

[Matthew Mosesohn (raytrac3r)]

The release file in .fuel-library.pkgs.ubuntu.neutron_vlan_ha (good):
http://paste.openstack.org/show/gNfcy5l6XHbJjkpIxBLg/
The release file in fuellib_review_pkgs_master_node (bad):
http://paste.openstack.org/show/lPN43dqN2dYqnzHiXZAK/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234304

[Dennis Dmitriev (ddmitriev)]

The issue happened because new packages are uploaded by fuel-qa tests before the Fuel admin node deployed and initialize auxiliary repositories with correct metadata. Thus, for Ubuntu, auxiliary repository gets empty metadata in 'Release' file and cannot be correctly processed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234304
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=a8c4492e12ce58badffa5e90caab6672c6c0e732
Submitter: Jenkins
Branch: master

commit a8c4492e12ce58badffa5e90caab6672c6c0e732
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/238099

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/238099
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Submitter: Jenkins
Branch: stable/7.0

commit e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242
    (cherry picked from commit a8c4492e12ce58badffa5e90caab6672c6c0e732)