[fuel-qa] Incorrect fuel-agent deb package version in master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy test

[Sergey Kulanov]

We need to clarify why do we have incorrect package version check in ironic bootstrap (check_package_version_injected_in_bootstraps) which leads to failed CI.

please check [1]:

2016-04-25 11:23:32,806 - INFO base_test_case.py:121 --
                                                       <<< 10. Verify fuel-agent version in ubuntu and ironic-bootstrap >>>
2016-04-25 11:23:32,945 - INFO utils.py:452 -- unpack active bootstrap
2016-04-25 11:23:47,841 - INFO utils.py:469 -- check package fuel-agent version injected in ubuntu bootstrap
2016-04-25 11:23:47,994 - INFO utils.py:478 -- package from review is 10.0.0-279.2.gerrit309645.3.git5fa99a9
2016-04-25 11:23:48,170 - INFO utils.py:486 -- injected package is 10.0.0-279.2.gerrit309505.1.git5fa99a9

so the correct package from patchset is:
10.0.0-279.2.gerrit309645.3.git5fa99a9

from where 10.0.0-279.2.gerrit309505.1.git5fa99a9 came from?

we need to investigate

[1]. https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/46/

[Alexander Gordeev (a-gordeev)]

In addition to that:

regular bootstrap image was built withstale fuel-agent package: 'fuel-agent': '10.0.0-1~u14.04+mos278'
but ironic-bootstrap was built with right fuel-agent package: 'fuel-agent': '10.0.0-279.2.gerrit309645.3.git5fa99a9'

[Nastya Urlapova (aurlapova)]

Infra team, as you should know, qa don't build any packages, we just use them in tests, so please check all working dirs on host system.

[Sergey Kulanov (skulanov)]

folks, it's about snapshoting IMO
please check builds:

31. [ok]
2016-04-22 18:42:23,466 - INFO utils.py:478 -- package from review is 10.0.0-279.2.gerrit309505.1.git5fa99a9
2016-04-22 18:42:23,628 - INFO utils.py:486 -- injected package is 10.0.0-279.2.gerrit309505.1.git5fa99a9

37. [failed]
2016-04-23 05:36:03,263 - INFO utils.py:478 -- package from review is 10.0.0-279.1.git5fa99a9
2016-04-23 05:36:03,441 - INFO utils.py:486 -- injected package is 10.0.0-279.2.gerrit309505.1.git5fa99a9

46. [failed]
2016-04-25 11:23:47,994 - INFO utils.py:478 -- package from review is 10.0.0-279.2.gerrit309645.3.git5fa99a9
2016-04-25 11:23:48,170 - INFO utils.py:486 -- injected package is 10.0.0-279.2.gerrit309505.1.git5fa99a9

[Sergey Kulanov (skulanov)]

since we restoring snapshot, we have "previous" packages from auxiliary repo, so we can:

1) clean auxiliary repo each time
2) do not use snapshoting
3) do snapshoting before uploading packages to auxiliary repo

[Artem Panchenko (apanchenko-8)]

Yep, Sergey is right, we have to clean repository before each rebuild, because new packages from CRs could have lower versions. But the main issue it that we add new packages to auxiliary repository after bootstrap image is rebuilt:

4. Rebuild bootstrap
```2016-04-25 10:29:23,632 - DEBUG helpers.py:357 -- Executing command: 'fuel-bootstrap build'
2016-04-25 10:34:14,830 - DEBUG ssh_manager.py:201 -- Host: 10.109.1.2
Command: 'fuel-bootstrap build '
Exit code: 0
STDOUT:
Bootstrap image 2f7c8316-626c-4fc5-8fff-2cccc54460a7 has been built: /tmp/2f7c8316-626c-4fc5-8fff-2cccc54460a7.tar.gz```

 9. Deploy the cluster
```2016-04-25 10:37:46,965 - DEBUG ssh_manager.py:201 -- Host: 10.109.1.2
Command: 'chmod 755 /tmp//regenerate_ubuntu_repo'
Exit code: 0
STDOUT:

STDERR:

2016-04-25 10:37:46,965 - DEBUG ssh_manager.py:104 -- SSH_MANAGER:Return existed connection for 10.109.1.2:22
2016-04-25 10:37:46,965 - DEBUG ssh_manager.py:105 -- SSH_MANAGER: Connections {('10.109.1.2', 22): <devops.helpers.helpers.SSHClient object at 0x7fa7b9601e10>, (u'10.109.1.3', 22): <devops.helpers.helpers.SSHClient object at 0x7fa7b9731790>, (u'10.109.1.4', 22): <devops.helpers.helpers.SSHClient object at 0x7fa7b96d9290>}
2016-04-25 10:37:46,966 - DEBUG helpers.py:357 -- Executing command: 'cd ~'
2016-04-25 10:37:47,043 - DEBUG helpers.py:357 -- Executing command: '/tmp//regenerate_ubuntu_repo /var/www/nailgun/ubuntu/auxiliary/ auxiliary'
2016-04-25 10:37:47,290 - INFO regenerate_repo.py:293 -- Local repository /var/www/nailgun/ubuntu/auxiliary/ has been updated successfully.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/309985

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309985
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=86dfc2d009d2150edd45b65624e700528b7eb30b
Submitter: Jenkins
Branch: master

commit 86dfc2d009d2150edd45b65624e700528b7eb30b
Author: Artem Panchenko <email address hidden>
Date: Mon Apr 25 17:10:59 2016 +0300

    Upload new packages before rebuilding bootstrap

    Packages assembled from CR were uploaded after cluster
    creation in step #9, so bootstrap image was built with
    old package. Now test uploads fresh packages before
    rebuilding bootstrap image.

    Also moved the step with package version verification
    before the step with cluster deployment and added
    possibility to remove old packages from target
    repository while uploading new files.

    Change-Id: Id768b87980fa112c1dbb82f886f390798ce4d8b0
    Closes-bug: #1574591

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310198

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310198
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=c51c39282b14890a7a05f4eb82f8a849e159a6c6
Submitter: Jenkins
Branch: stable/mitaka

commit c51c39282b14890a7a05f4eb82f8a849e159a6c6
Author: Artem Panchenko <email address hidden>
Date: Mon Apr 25 17:10:59 2016 +0300

    Upload new packages before rebuilding bootstrap

    Packages assembled from CR were uploaded after cluster
    creation in step #9, so bootstrap image was built with
    old package. Now test uploads fresh packages before
    rebuilding bootstrap image.

    Also moved the step with package version verification
    before the step with cluster deployment and added
    possibility to remove old packages from target
    repository while uploading new files.

    Change-Id: Id768b87980fa112c1dbb82f886f390798ce4d8b0
    Closes-bug: #1574591
    (cherry picked from commit 86dfc2d009d2150edd45b65624e700528b7eb30b)

[Alexander Gordeev (a-gordeev)]

recent job https://ci.fuel-infra.org/job/master.fuel-agent.pkgs.ubuntu.review_fuel_agent_ironic_deploy/56/ failed with

AssertionError: The new package 10.0.0-280.2.gerrit310244.1.git0c100a3 wasn't injected in bootstrap

[Sergey Kulanov (skulanov)]

@Alexander CI-team is going to update fuel-qa version for tests on fuel-ci

[Dmitry Kaigarodеsev (dkaiharodsev)]

fuel-qa on for master jobs has been updated https://ci.fuel-infra.org/view/devops/job/devops.master.create-env/1069/console
up to revision c8981c5196f068d87e6f165d730ddf4700a8e6ba

fuel-qa for mitaka branch has been updated https://ci.fuel-infra.org/view/devops/job/devops.mitaka.create-env/28/console up to revision 21584924af9cc5ac9d1612a4a43ccc033660f1ad

[Nastya Urlapova (aurlapova)]

Since Dima's comment, it was moved to "Fix Released".