fuellib_review_pkgs_master_node does not use packages built as part of change

Bug #1502242 reported by Ryan Moe on 2015-10-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Dennis Dmitriev
7.0.x
Critical
Dennis Dmitriev

Bug Description

This jobs builds new packages but they are not used when deploying environments. See this run [0] for an example.

A new fuel-ha-utils package was built ( fuel-ha-utils_8.0.0-7290.2.gerrit225941.6.gitea9fe4b_all.deb) but in the failed deployment the controller had 8.0.0-7288.1.gitacfcfd2 installed. The package built by the job included the correct changes. I was not able to look at the contents of the installed package but because the test failed I can safely assume they did not include the necessary changes. The job was triggered by this review [1].

[0] https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4775/
[1] https://review.openstack.org/#/c/225941/

Ryan Moe (rmoe) on 2015-10-02
Changed in fuel:
milestone: none → 8.0
assignee: nobody → Fuel QA Team (fuel-qa)
Sergey Kulanov (skulanov) wrote :

Some update, from discussion with Artem:

No aux repo in pinning rules:

root@node-1:~# cat /etc/apt/preferences.d/*
Package: *
Pin: release
Pin-Priority: 1150Package: *
Pin: release o=Mirantis,a=mos8.0-holdback,l=mos8.0,n=mos8.0-holdback
Pin-Priority: 1100Package: *
Pin: release n=mos8.0,o=Mirantis,l=mos8.0,a=mos8.0
Pin-Priority: 1050Package: *
Pin: release o=Mirantis,n=mos8.0-security,a=mos8.0-security,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release a=mos8.0-updates,o=Mirantis,n=mos8.0-updates,l=mos8.0
Pin-Priority: 1050Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,a=trusty-updates,n=trusty,v=14.04
Pin-Priority: 1001Package: *
Pin: release l=Ubuntu,o=Ubuntu,n=trusty,v=14.04,a=trusty-security

So built packages in aux repo, has default pinning rules:

root@node-1:~# apt-cache policy fuel-misc
fuel-misc:
  Installed: 8.0.0-7288.1.gitacfcfd2
  Candidate: 8.0.0-7288.1.gitacfcfd2
  Version table:
     8.0.0-7292.1.git84cdfdf 0
        500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
 *** 8.0.0-7288.1.gitacfcfd2 0
       1050 http://10.109.5.2:8080/2015.1.0-8.0/ubuntu/x86_64/ mos8.0/main amd64 Packages
        100 /var/lib/dpkg/status

Sergey Kulanov (skulanov) wrote :

Set to critical, since we have CI impact

Changed in fuel:
importance: Undecided → Critical
Sergey Kulanov (skulanov) wrote :

Critical for 7.0 since the same issue:

fuel-snapshot-2015-10-09_04-01-08/node-1.test.domain.local/commands/ubuntu_repo_list.txt

[node-1.test.domain.local] out: 1100 http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/ mos7.0-holdback/main amd64 Packages
[node-1.test.domain.local] out: release o=Mirantis,a=mos7.0-holdback,n=mos7.0,l=mos7.0,c=main
[node-1.test.domain.local] out: origin mirror.fuel-infra.org
[node-1.test.domain.local] out: 500 http://10.109.0.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.0.2
[node-1.test.domain.local] out: Pinned packages:
[node-1.test.domain.local] out:

Changed in fuel:
status: New → Confirmed
Maksym Strukov (unbelll) on 2015-10-12
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Fuel Library Team (fuel-library)
Maksym Strukov (unbelll) wrote :

Ryan's run [0] was analyzed. In agree with snapshot, all three nodes has correct record for aux repo in astute.yaml:
'''
  - name: Auxiliary
    section: main
    uri: http://10.109.5.2:8080/ubuntu/auxiliary/
    priority: 1150
    suite: auxiliary
    type: deb
'''

But after provisioning in /etc/apt/preferences.d/* aux repo has wrong priority:
'''
500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
release c=main
origin 10.109.5.2
'''

Reassigned to fuel-lib for further investigation, because it's all ok on fuel-qa side.

---
[0] https://ci.fuel-infra.org/job/fuellib_review_pkgs_master_node/4775/artifact/logs/4775/fail_error_deploy_neutron_tun-fuel-snapshot-2015-10-02_14-17-34.tar.xz

Matthew Mosesohn (raytrac3r) wrote :

On my locally deployed env, I see
root@node-2:~# apt-cache policy | grep -A3 aux
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/restricted amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2
1150 http://10.20.0.2:8080/2015.1.0-8.0/ubuntu/auxiliary/ auxiliary/main amd64 Packages
     release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=restricted
     origin 10.20.0.2

From that snapshot from the CI job, it's:
[mmosesohn@localhost commands]$ grep -A3 aux ubuntu_repo_list.txt
[node-1.test.domain.local] out: 500 http://10.109.5.2:8080/ubuntu/auxiliary/ auxiliary/main amd64 Packages
[node-1.test.domain.local] out: release c=main
[node-1.test.domain.local] out: origin 10.109.5.2

The regenerate-ubuntu-repo script is generating the wrong Release metadata.
currently it has :
release c=main

but it needs to have:
release o=Mirantis,a=auxiliary,n=auxiliary,l=auxiliary,c=main

The pinning file depends on generating the correct Release metadata

Matthew Mosesohn (raytrac3r) wrote :

The release file in .fuel-library.pkgs.ubuntu.neutron_vlan_ha (good):
http://paste.openstack.org/show/gNfcy5l6XHbJjkpIxBLg/
The release file in fuellib_review_pkgs_master_node (bad):
http://paste.openstack.org/show/lPN43dqN2dYqnzHiXZAK/

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Dennis Dmitriev (ddmitriev)

Fix proposed to branch: master
Review: https://review.openstack.org/234304

Changed in fuel:
status: Confirmed → In Progress
Dennis Dmitriev (ddmitriev) wrote :

The issue happened because new packages are uploaded by fuel-qa tests before the Fuel admin node deployed and initialize auxiliary repositories with correct metadata. Thus, for Ubuntu, auxiliary repository gets empty metadata in 'Release' file and cannot be correctly processed.

Reviewed: https://review.openstack.org/234304
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=a8c4492e12ce58badffa5e90caab6672c6c0e732
Submitter: Jenkins
Branch: master

commit a8c4492e12ce58badffa5e90caab6672c6c0e732
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/238099
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Submitter: Jenkins
Branch: stable/7.0

commit e5e5c1ed43cc1d2544cf215d4bd743eb6e165405
Author: Dennis Dmitriev <email address hidden>
Date: Tue Oct 13 18:39:58 2015 +0300

    Separate uploading packages for patching

    - RPM packages are uploaded after Fuel admin node bootstrapped,
      as previous;
    - DEB packages are uploaded after Fuel admin node deployed and
      'auxiliary' repository initialized.

    Change-Id: I5c423529e175ef6ddfc9ac5f5e2359e1e0b6d32f
    Closes-Bug:#1502242
    (cherry picked from commit a8c4492e12ce58badffa5e90caab6672c6c0e732)

Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-qa
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers