Stop using hard coded list of packages during creating IBP

[Albert Syriy]

The list of packages are using for building IBP images is hard coded now. See for details [1]. [2]
It disable user to add a new package without changing code of the fuel-agent.
It's a sort of blocker for adding/removing drivers/kernel modules.
Please provide a way for managing package list (adding/removing the packages) by using a config file.

[1] https://github.com/openstack/fuel-agent/blob/9bc9b160bac48b9332a38b78e62330622d6c2750/contrib/fuel_bootstrap/fuel_bootstrap_cli/fuel_bootstrap/settings.yaml.sample#L18-l41

[2] https://github.com/openstack/fuel-agent/blob/95fe335f2451703e68318787196a1ea4648f94e3/fuel_agent/drivers/nailgun.py#L669-L719

[Aleksey Zvyagintsev (azvyagintsev)]

# fuel-bootstrap help build
  --no-default-packages
                        Do not append default packages
  --package PKGNAME The option can be given multiple times, all specified
                        packages and their dependencies will be installed.

[Arthur Svechnikov (asvechnikov)]

You can use CLI tool fuel-bootstrap as Aleksey mentioned above or you can manage packages by editing of /etc/fuel-bootstrap-cli/fuel_bootstrap_cli.yaml

[Ihor Kalnytskyi (ikalnytskyi)]

Arthur, why did you move this bug to Invalid? If I understand things right, it's still valid for IBP case - you can't pass custom set of packages for Ubuntu provisioning image.

[Ivan Ponomarev (ivanzipfer)]

I agree that [2] should be moved to configuration file.
[1] settings.yaml - looks like normal conf file

[Alexander Gordeev (a-gordeev)]

> I agree that [2] should be moved to configuration file.

IBP images produced on per cluster basis. Therefore, I believe it's a matter for cluster setting and should be handled at nailgun side.

[Oleg S. Gelbukh (gelbuhos)]

+1 for handling packages list as a cluster attribute, and in future add an ability to customize the list of packages per node.

This should help solve issues with handling legacy environments with upgraded Fuel Admin node: we will know exactly which packages were used to build IBP image in the environment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/273974

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/273974
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=4ee42effe27694bd231663e3d0f10c0c42877177
Submitter: Jenkins
Branch: master

commit 4ee42effe27694bd231663e3d0f10c0c42877177
Author: Alexander Saprykin <email address hidden>
Date: Fri Jan 29 11:09:41 2016 +0100

    Move IBP packages list to release attributes

    List of ubuntu packages that are installed during provisioning was
    hardcoded in fuel-agent.
    Add list of packages to cluster attributes, so it
    can be configured in UI or CLI.

    Change-Id: I858cebd868d2e679837bb129bdd5e1194fdf3a32
    Closes-Bug: #1538140

[Aleksey Zvyagintsev (azvyagintsev)]

Guys, is it any chance to backport in to 8.0 ?

[Albert Syriy (asyriy)]

We really need it in 8.0 to let customers manipulate with list of packages installed on IBP (Ubuntu).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/298632

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/302764

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Saprykin (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/298632

[Alexander Saprykin (cutwater)]

Closed as Won't fix since it's not a customer-found bug, so it will not be backported to 8.0

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Saprykin (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/302764
Reason: Not a customer-found bug. Won't be included into 8.0

[Ilya Kharin (akscram)]

I disagree how it was implemented and merged into the 9.0 release. In this release new type for data, such as a list of strings, was implemented [1], it is possible to use text_list instead of text. I suggest to use the text_list type as a more appropriate format for such data.

[1] https://review.openstack.org/#/c/284053/

[Ihor Kalnytskyi (ikalnytskyi)]

Ilya,

If I'm not mistaken, It was implemented much before "text_list" was introduced. If you feel like it's better to use "text_list", please feel free to provide a patch.

[Volodymyr Shypyguzov (vshypyguzov)]

Hi, guys. I have one question regarding IBP image customizing

Should Fuel rebuild image after changing initial packages list?

Steps to reproduce:
1. Create environment
2. Provision one node
3. Add new package to the initial packages list
4. Provision one more node

As I see now Fuel uses image built on step 2. Is this expected behavior?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/309054

[Alexander Gordeev (a-gordeev)]

@Volodymyr Shypyguzov (vshypyguzov)

> As I see now Fuel uses image built on step 2. Is this expected behavior?

Yes, that is totally expected behaviour. If images for the current environment already presented they won't be rebuild anyway. To force rebuild of images, you just need to delete them.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309054
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7f6977c52b1ec7336933f848a8c351b936dbbd00
Submitter: Jenkins
Branch: master

commit 7f6977c52b1ec7336933f848a8c351b936dbbd00
Author: Volodymyr Shypyguzov <email address hidden>
Date: Thu Apr 21 17:17:46 2016 +0300

    Add tests for IBP

    Add tests for Image Based Provisioning
    +Add test default packages are installed after node provisioning
    +Add test adding package to list installs it
    +Add test removing package from list doesn't install it
    +Add test adding wrong package name breaks provisioning

    Related-bug: #1538140

    Change-Id: I71b89b7e14547031eb207f89fab0cc6bab788330

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/316686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/316686
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=ef9698cf30536953cb2e5da6e4d4fee3aa851e06
Submitter: Jenkins
Branch: stable/mitaka

commit ef9698cf30536953cb2e5da6e4d4fee3aa851e06
Author: Volodymyr Shypyguzov <email address hidden>
Date: Thu Apr 21 17:17:46 2016 +0300

    Add tests for IBP

    Add tests for Image Based Provisioning
    +Add test default packages are installed after node provisioning
    +Add test adding package to list installs it
    +Add test removing package from list doesn't install it
    +Add test adding wrong package name breaks provisioning

    Related-bug: #1538140

    Change-Id: I71b89b7e14547031eb207f89fab0cc6bab788330

[Nastya Urlapova (aurlapova)]

Test for IBP are passed on ISO #503:
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.test_ibp/22/testReport/(root)/
check_ibp_add_package
check_ibp_add_wrong_package
check_ibp_default_package_list 13 min
check_ibp_remove_package