[IBP] Re-write in Python image build script

[Vladimir Kozhukalov]

Currently it written in shell, needs to be re-written in python.
Reasons:
1) logging
2) working with input data
3) convenience
4) maintainability

[Ihor Kalnytskyi (ikalnytskyi)]

Since we're going to work closely with OS, I suggest to use python's sh library. http://amoffat.github.io/sh/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/167617

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/169413

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/170551

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/174297

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167705
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f1e880689226ff5ca7c774ef18b0dc0be41fedec
Submitter: Jenkins
Branch: master

commit f1e880689226ff5ca7c774ef18b0dc0be41fedec
Author: Alexander Gordeev <email address hidden>
Date: Wed Mar 25 20:22:25 2015 +0300

    [IBP] Add build_utils

    Build utils is providing all necessary low-level functions to build
    target image.

    Related-Bug: #1433193
    Partially implements: blueprint ibp-build-ubuntu-images
    Change-Id: I8d89377fb38073968a580f7c8e3d04e61ce8ed60

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/167617
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=f284bd9f9710238408d3ab7b60fb7f8167c15b0e
Submitter: Jenkins
Branch: master

commit f284bd9f9710238408d3ab7b60fb7f8167c15b0e
Author: Vladimir Kozhukalov <email address hidden>
Date: Wed Mar 25 17:13:52 2015 +0300

    IBP: Improved data parsing flow

    It is better when manager does not need to
    care about calling data driver parse method but
    instead manager just instantiates data driver
    with raw data and then asks data driver for
    ready to use objects.

    Even more, it is necessary to encapsulate this logic
    into a data driver itself because which schemes we
    actually need (PartitionScheme, ConfigdriveScheme)
    depend on the manager's do_* method we use. Some
    do_* methods don't even need PartitionScheme or
    ConfigdriveScheme and a user who calls for example
    do_reboot method does not assume she needs to
    provide fuel-agent with partitioning data.

    Related-bug: #1433193
    Implements: blueprint ibp-build-ubuntu-images
    Change-Id: I79b73d63c1ecb1dd0af24bb446003cb6585d206f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169413
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=005c9c9c6e1515c11cc4a94cf2597bbb6f797307
Submitter: Jenkins
Branch: master

commit 005c9c9c6e1515c11cc4a94cf2597bbb6f797307
Author: Vladimir Kozhukalov <email address hidden>
Date: Tue Mar 31 19:25:20 2015 +0300

    IBP: Added driver and objects for building images

    As far as building of OS images is nothing more than
    just a stage of the whole OS installing procedure
    it is sounds rational to implement this in terms
    of fuel-agent. Besides, we already have plenty of utilities
    which could be useful during building of images.
    And some tasks are the same like pre-configuring
    some files inside target OS.

    Related-bug: #1433193
    Implements: blueprint ibp-build-ubuntu-images
    Change-Id: I3fadfb16e06e4ee16926da29b7b83ca005500698

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/170551
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=32902288a47f1e842a4dfb03aca51743f72e1822
Submitter: Jenkins
Branch: master

commit 32902288a47f1e842a4dfb03aca51743f72e1822
Author: Vladimir Kozhukalov <email address hidden>
Date: Fri Apr 3 18:59:10 2015 +0300

    IBP: install fuel-agent into mcollective docker

    We implemented target OS image building
    in terms of fuel-agent. So, we need this package
    to be installed into mcollective docker container.
    fuel-agent provides build_image executable entry
    point.

    Change-Id: Ic7d10fead8423cdb8fd33c3244e0670d4ed19f4f
    Implements: blueprint ibp-build-ubuntu-images
    Related-bug: #1433193

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/174297
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=a7885a2ec0a1534b74c0ae0f2390dc1081325571
Submitter: Jenkins
Branch: master

commit a7885a2ec0a1534b74c0ae0f2390dc1081325571
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Apr 16 13:35:04 2015 +0300

    IBP: Changed image metadata format

    New metadata format consists of tree parts
    1) images metadata (md5 sums, sizes, filenames, formats)
    2) repos metadata (which repos were used
    to build this set of images)
    3) packages metadata (just a list of packages
    installed into an image)

    The motivation behind is to split the image abstraction
    (which is not always just OS image) from repos and packages
    abstractions.

    Change-Id: I5ce3e291a3ad7a1b7012b5c504ecdc97dd4023e2
    Implements: blueprint ibp-build-ubuntu-images
    Related-bug: #1433193

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/175847

[Anastasia Palkina (apalkina)]

Verified on ISO #338

"build_id": "2015-04-22_22-54-31", "build_number": "338", "release_versions": {"2014.2.2-6.1": {"VERSION": {"build_id": "2015-04-22_22-54-31", "build_number": "338", "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "9b26cb98e7ceb905fe85b744eee2f808f7e4737b", "nailgun_sha": "4972120d5a5d41efe9be2d10b81329adf3258a78", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "2b311b3b82a1e2df1dc3484a0f37e282273cd988", "astute_sha": "5be3506e6a4f8b7f0a858db2e162dabcc07347df", "fuel-ostf_sha": "dcda5a869d1780efc265b6f6b7b8c14c8324a68b", "release": "6.1", "fuelmain_sha": "2f8be90209e709810674f73abb02bb54a700ce5e"

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladimir Kozhukalov (<email address hidden>) on branch: master
Review: https://review.openstack.org/175847