Bug #1641157 “support multiple -p options in disk-image-create” : Bugs : diskimage-builder

Oliver Walsh (owalsh) on 2016-11-11

description:

updated

Oliver Walsh (owalsh) on 2016-11-11

Changed in diskimage-builder:
assignee:	nobody → Oliver Walsh (owalsh)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-11: Fix proposed to diskimage-builder (master)

#1

Fix proposed to branch: master
Review: https://review.openstack.org/396702

Changed in diskimage-builder:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-17: Fix merged to diskimage-builder (master)

#2

Reviewed: https://review.openstack.org/396702
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=ae66b64c34ea4209e09adf804c8cd8446d7ae5db
Submitter: Jenkins
Branch: master

commit ae66b64c34ea4209e09adf804c8cd8446d7ae5db
Author: Oliver Walsh <email address hidden>
Date: Fri Nov 11 17:03:50 2016 +0000

In disk-image-create, append to INSTALL_PACKAGES instead of clobbering.

This allows -p <packagelist> to be used multiple times.

Change-Id: Iabe43982e1606c7ca963a1dd3b23ba47d148ae38
Closes-Bug: #1641157

Changed in diskimage-builder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-18: Fix proposed to diskimage-builder (feature/v2)

#3

Fix proposed to branch: feature/v2
Review: https://review.openstack.org/399347

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-18: Fix merged to diskimage-builder (feature/v2)

#4

Download full text (14.6 KiB)

Reviewed: https://review.openstack.org/399347
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=7d5afecfd9e2ea5249ea7f12eba74c77c1abd7e6
Submitter: Jenkins
Branch: feature/v2

commit 8d7362aa9b4a9d2430de7959d4ee4c78429464c0
Author: d.marlin <email address hidden>
Date: Thu Nov 17 03:36:54 2016 -0500

Change path for dnf arch override so basearch is not overwritten.

    After writing the basearch value to /etc/dnf/vars/basearch the
    arch value was overwriting the same file. This appears to be
    incorrect, so changing it to write /etc/dnf/vars/arch, which
    matches the subsequent 'yum' code paths.

Change-Id: I5da54f03224c11f9e286f16b68533936c4174c2a

commit fb8cf95b6fb0820867ebfd262c38d2cae5ab43fa
Author: Ben Nemec <email address hidden>
Date: Wed Nov 16 16:27:24 2016 -0600

Disable all repos in os-refresh-config too

    This change was made for pre-install so it applies during the
    image build, but wasn't applied to the os-refresh-config script
    that would run after deployment. The same problems apply there,
    so we should do the same thing.

Change-Id: I4b8534cc9586eeb588b5c358550e76e27d40556a
Closes-Bug: 1629922

commit a1f57b8cad0717090eb12014d7847a35bd0563a3
Author: Markos Chandras <email address hidden>
Date: Mon Oct 31 21:43:47 2016 +0000

lib: common-functions: Fix tmpfs umounting

    It has been observed that some chroot operations spawn additional
    processes which rely on chroot files. More specifically, zypper, uses
    gpg-agent to import and validate gpg keys for its repositories. This
    gpg-agent process may stay alive for longer which prevents unmounting of
    the tmpfs directory since the gpg-agent process still uses libraries etc
    which were present in the chroot. We try to solve this by using walking
    all the pids in /proc to find out the running processes in the chroot and
    kill them gracefully. If that fails for whatever reason, then we simply
    keep trying to umount the tmpfs directory before we give up.

    The gpg-agent process usually terminates soon after its home directory
    disappears but on fast systems we can reach the 'umount tmpfs' point
    before gpg-agent terminates by itself. The solution is generic enough so
    other 'chroot processes' can also be handled appropriately.

Change-Id: Iccf332678c79266113e76f062884fc5ee79e515d

commit e88d6b37df5a210ad15da24403274914f737956e
Author: Noam Angel <email address hidden>
Date: Mon Oct 31 11:54:58 2016 +0200

add support for SUSE in dhcp-all-interfaces

    This patch will add support for SUSE network scripts,
    network script in SUSE saved under "/etc/sysconfig/network/ifcfg-*"
    see: https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_basicnet_manconf.html

Change-Id: I87ac2e327cee4945c15da9f2e4adc0a8b7650712

commit d07d7ed15d54c37448d5be67c4e4707ba19edac0
Author: Noam Angel <email address hidden>
Date: Tue Nov 1 16:19:44 2016 +0200

simplify ARCH param for rhel/centos param can be x86_64 and amd64

for fedora/rhel/centos the main supported ARCH is x86_64. This patch allow
to call diskima...

Reviewed:  https://review.openstack.org/399347
Committed: https://git.openstack.org/cgit/openstack/diskimage-builder/commit/?id=7d5afecfd9e2ea5249ea7f12eba74c77c1abd7e6
Submitter: Jenkins
Branch:    feature/v2

commit 8d7362aa9b4a9d2430de7959d4ee4c78429464c0
Author: d.marlin <dmarlin@redhat.com>
Date:   Thu Nov 17 03:36:54 2016 -0500

Change path for dnf arch override so basearch is not overwritten.
    
    After writing the basearch value to /etc/dnf/vars/basearch the
    arch value was overwriting the same file.  This appears to be
    incorrect, so changing it to write /etc/dnf/vars/arch, which
    matches the subsequent 'yum' code paths.
    
    Change-Id: I5da54f03224c11f9e286f16b68533936c4174c2a

commit fb8cf95b6fb0820867ebfd262c38d2cae5ab43fa
Author: Ben Nemec <bnemec@redhat.com>
Date:   Wed Nov 16 16:27:24 2016 -0600

Disable all repos in os-refresh-config too
    
    This change was made for pre-install so it applies during the
    image build, but wasn't applied to the os-refresh-config script
    that would run after deployment.  The same problems apply there,
    so we should do the same thing.
    
    Change-Id: I4b8534cc9586eeb588b5c358550e76e27d40556a
    Closes-Bug: 1629922

commit a1f57b8cad0717090eb12014d7847a35bd0563a3
Author: Markos Chandras <mchandras@suse.de>
Date:   Mon Oct 31 21:43:47 2016 +0000

lib: common-functions: Fix tmpfs umounting
    
    It has been observed that some chroot operations spawn additional
    processes which rely on chroot files. More specifically, zypper, uses
    gpg-agent to import and validate gpg keys for its repositories. This
    gpg-agent process may stay alive for longer which prevents unmounting of
    the tmpfs directory since the gpg-agent process still uses libraries etc
    which were present in the chroot. We try to solve this by using walking
    all the pids in /proc to find out the running processes in the chroot and
    kill them gracefully. If that fails for whatever reason, then we simply
    keep trying to umount the tmpfs directory before we give up.
    
    The gpg-agent process usually terminates soon after its home directory
    disappears but on fast systems we can reach the 'umount tmpfs' point
    before gpg-agent terminates by itself. The solution is generic enough so
    other 'chroot processes' can also be handled appropriately.
    
    Change-Id: Iccf332678c79266113e76f062884fc5ee79e515d

commit e88d6b37df5a210ad15da24403274914f737956e
Author: Noam Angel <noama@mellanox.com>
Date:   Mon Oct 31 11:54:58 2016 +0200

add support for SUSE in dhcp-all-interfaces
    
    This patch will add support for SUSE network scripts,
    network script in SUSE saved under "/etc/sysconfig/network/ifcfg-*"
    see: https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_basicnet_manconf.html
    
    Change-Id: I87ac2e327cee4945c15da9f2e4adc0a8b7650712

commit d07d7ed15d54c37448d5be67c4e4707ba19edac0
Author: Noam Angel <noama@mellanox.com>
Date:   Tue Nov 1 16:19:44 2016 +0200

simplify ARCH param for rhel/centos param can be x86_64 and amd64
    
    for fedora/rhel/centos the main supported ARCH is x86_64. This patch allow
    to call diskimage-builder with the above distro's with param ARCH=x86_64,
    And also retain same behaiver when call with ARCH=amd64 as it translate
    anyway to x86_64. Doing so wil simplify user expirience.
    
    Change-Id: I229e0912434109b1b48a030bd35ad8dc1096a629

commit 5d9d3d5cf0c7ab79a6cd32d43375fa683ae87609
Author: Saverio Proto <saverio.proto@switch.ch>
Date:   Mon Nov 14 15:32:08 2016 +0100

debian: install dialog package
    
    Without the dialog package is not possible
    to properly use an interactive frontend.
    
    debconf will print the following errors:
    
    debconf: unable to initialize frontend: Dialog
    debconf: (No usable dialog-like program is installed,
    so the dialog based frontend cannot be used. at
    /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)
    
    Change-Id: I0c7142f717cacf7437dbac1e1696f39b00cb4c49

commit 79c27199f2ce11c579efa0cfce7ee2c3443ead89
Author: Ben Nemec <bnemec@redhat.com>
Date:   Fri Nov 11 14:30:21 2016 -0600

Install lsb package by map name instead of package name
    
    We have a pkg-map entry for lsb_release, but in package-installs.yaml
    we refer to the actual package name instead.  This will happen to
    work on Red Hat platforms, but it's actually wrong.
    
    Change-Id: Idb248f96e75fa1090422fa08e5fbb2385cc1f517

commit ae66b64c34ea4209e09adf804c8cd8446d7ae5db
Author: Oliver Walsh <owalsh@redhat.com>
Date:   Fri Nov 11 17:03:50 2016 +0000

In disk-image-create, append to INSTALL_PACKAGES instead of clobbering.
    
    This allows -p <packagelist> to be used multiple times.
    
    Change-Id: Iabe43982e1606c7ca963a1dd3b23ba47d148ae38
    Closes-Bug: #1641157

commit 7e60540c0d8d02c976f7890831e103584501c27c
Author: Ian Wienand <iwienand@redhat.com>
Date:   Thu Nov 10 14:22:17 2016 +1100

Cleanup yumdownloader repos
    
    yumdownloader has to have all the repo XML files, etc, which adds up
    to a not totally insignificant 150MiB or so.  Currently we're leaking
    this directory for every build, which adds up on regualar builders
    like nodepool.
    
    Isolate the call with a separate TMPDIR so we can clean it up after
    the initial download.
    
    Change-Id: Ic65e8ca837cc76b7a1bb9f83027b4a5bdd270f75

commit 0f742ce24a39706545333a2cad983bc0a7a11b9e
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Nov 9 04:14:55 2016 +0000

Updated from global requirements
    
    Change-Id: I4b8e6f6fdf78706dbb735e25195a1f83413ab3fd

commit c7219a5a605e68f0ca46f4f1c130b602e25be073
Author: Luca Lorenzetto <lorenzetto.luca@gmail.com>
Date:   Mon Oct 10 16:00:16 2016 +0200

Avoid disabling rhel-7-server-rh-common-rpms
    
    while using disk-image-builder for building overcloud images for TripleO
    using RDO, this repository is (in my opinion) wrongly disabled because
    contains certain dependencies needed by RDO packages.
    Example: python-cheetah is required for python-nova, but is not
    available through RDO repository but only from
    rhel-7-server-rh-common-rpms
    
    Closes-Bug: #1638938
    Change-Id: I76824c8ec02590397f1ff1d4f177ad061c7bf441
    Signed-off-by: Luca Lorenzetto <lorenzetto.luca@gmail.com>

commit 435f52a7d1506b87d6b650adcdde35a36c242157
Author: Markos Chandras <mchandras@suse.de>
Date:   Wed Nov 2 22:05:05 2016 +0000

elements: zypper-minimal: Mount common pseudo filesystems
    
    Mount all the usual /dev /sys /proc pseudo filesystems during the
    root.d phase in order to make sure they are available for the rpm
    post-installation phases.
    
    Change-Id: I28221debf1036d9eb5137161757eb30811eafab1

commit bc66298937c55321874eb599aeddde25ce125dd3
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Wed Nov 2 15:31:00 2016 +0000

Updated from global requirements
    
    Change-Id: Ib41e9cf05743c05fb299f84516d9179ea41e3f46

commit d5225055ef2403b461083f5d76c67bc8410539a7
Author: Derek Higgins <derekh@redhat.com>
Date:   Wed Nov 2 11:01:15 2016 +0000

dhcp-all-interfaces: support Centos/RHEL 6
    
    On Centos and RHEL 6 the init system is upsart but but networking is using
    sysv compatabiliy and a code path the handle this situation.
    
    We can't use DISTRO_NAME because the centos-minimal element sets it to
    centos for CentOS 7 but the centos element sets it to centos for CentOS 6.
    
    Change-Id: Ib8e33ed78b3d6a5737eb7449bccef2d33f72b131
    Closes-Bug: #1638527

commit 290925bd5e0470f69357a8ff13e423ba628c49bb
Author: Markos Chandras <mchandras@suse.de>
Date:   Mon Oct 31 23:18:28 2016 +0000

elements: zypper-minimal: Refresh repositories after adding the cache
    
    The refresh operation must happen after the cache has been added in
    order to ensure that whatever is in the cache is still relevant to
    the current build and we are not using stale packages.
    
    Change-Id: Iafd718e9738f85b8c235806c027665730f44d89b

commit 348a6b337a18f4666ed99a387d20ed83c1aa86c8
Author: Markos Chandras <mchandras@suse.de>
Date:   Mon Oct 31 23:15:46 2016 +0000

elements: opensuse-minimal: Add support for building Tumbleweed images
    
    Add DIB_RELEASE=Tumbleweed option in order to build openSUSE Tumbleweed
    images
    
    Change-Id: I44cc04ef5a993c1a7f0078e4161888b52995f247

commit 2854f4063bd2a6dcdb6fa5fab93aa56857e47b59
Author: Bruno Cornec <bruno.cornec@hpe.com>
Date:   Fri Oct 28 19:38:50 2016 +0200

Fix ironic-python-agent image not loading vfat mod
    
    Closes-Bug: 1589450
    Load the vfat driver as a Pre Exec action for systemd before starting
    ironic-python-agent in order to allow reading of parameters.txt file
    required for the ironic-python-agent to find its configuration.
    
    Change-Id: Ibf74dd1b2678ea76e0676711a7aa5ba6b88d5421

commit 3b855faf5f5a27e2344f0185954f125e7baf931a
Author: nizam <abdul.nizamuddin@nectechnologies.in>
Date:   Fri Oct 28 19:51:55 2016 +0530

Don't include openstack/common in flake8 exclude list
    
    The directory openstack/common was used to keep codes from
    oslo-incubator, we have retired oslo-incubator, so don't use
    this directory any more
    
    Change-Id: Icd13d32674b117626fbecdfdec2881463a87ad5d

commit 51f982e3765705a619ec59671911163012265bea
Author: nizam <abdul.nizamuddin@nectechnologies.in>
Date:   Fri Oct 28 19:48:33 2016 +0530

Drop MANIFEST.in - it's not needed by pbr
    
    diskimage-builder already uses PBR:-
    setuptools.setup(
        setup_requires=['pbr>=1.8'],
        pbr=True)
    
    This patch removes `MANIFEST.in` file as pbr generates a
    sensible manifest from git files and some standard files
    and it removes the need for an explicit `MANIFEST.in` file.
    
    Change-Id: Iad8b724c1d1e0ef358606b13da4569d881ec740a

commit 4095046a25f94a06e44368f4f9726e9347f15e11
Author: nizam <abdul.nizamuddin@nectechnologies.in>
Date:   Fri Oct 28 19:46:02 2016 +0530

Changed the home-page of diskimage-builder in setup.cfg
    
    Instead of pointing to
    https://git.openstack.org/cgit/openstack/diskimage-builder, the
    homepage has been changed to point to the homepage of
    diskimage-builder.
    
    Change-Id: I795a5fd343ff7a06bfe43ae4a76dc24136db99a2

commit e531980a14d55ecf437188fb92fc332a82c2d537
Author: Ian Wienand <iwienand@redhat.com>
Date:   Wed Oct 19 09:34:54 2016 +0000

Remove RedHat grub workaround install
    
    AFAICT this is no longer necessary.  I've tested minimal and image
    builds and they seem to work.
    
    The original problem seems to be with installing the package in the
    chroot, although it was never quite clear it ever affected the Red Hat
    path.
    
    This code is currently broken (see
    I884cb1e78ad8c31d985f3fc94a58091b993edd7d).  This is proposed as an
    alternative to I74eed074494134334d5e49042bb5214bd0dd7339.
    
    Related-Bug: #1627000
    
    Change-Id: Iafe3611f4eec3c6357587a6cae6a30a261686ead

commit 6fb658a5f1bbc38baa07bb2f51acc6528b8c017a
Author: Ian Wienand <iwienand@redhat.com>
Date:   Mon Oct 24 11:17:28 2016 +1100

Don't log datestamp by default in functional tests
    
    We're getting double time-stamps in the console log of upstream jobs.
    Move the logging of a prefix datestamp into a "-t" option to retain
    the status quo prior to Id9ea5131f0026c292ca6453ba2c80fe12c47f808 (we
    could, of course, do it the other way and turn if off in the jobs, but
    since we didn't have it before...)
    
    While poking, make the time-stamp consistent and always prefixed if -t
    is turned on.
    
    Also, it seems the parallel options got a bit of sync with what got
    merged.  Add "-j" documentation and remove unused "p" option.
    
    Change-Id: Ic7c2ebeca3f9d5784cac59505b6e6181151f5805

commit 9e392f56b0b78f60bc2598c3cd99f8d7061cc550
Author: Ian Wienand <iwienand@redhat.com>
Date:   Thu Oct 20 13:51:16 2016 +1100

Don't set tracing in environment files
    
    Because environment files are sourced into the current environment,
    they shouldn't be setting global settings like tracing else they
    affect every preceeding import.  This is quite confusing when only
    half your imports are traced in the logs, because it was either turned
    on, or off, by a preceeding environment import.
    
    There is a corresponding dib-run-parts change in
    I29f7df1514aeb988222d1094e8269eddb485c2a0 that will greatly increase
    debugability for environment files by deliberately logging what files
    are sourced and consistently turning on tracing around their import.
    
    This isn't strictly necessary (since dib-run-parts with the prior
    change will just turn tracing off after import anyway) but it's a
    decent cleanup for consistency.  A bare-minimum dib-lint check is
    added.  Documentation is updated.
    
    Change-Id: I10f68be0642835a04af7e5a2bc101502f61e5357

commit 3d44a08c53cc9280b33e83fcec8766cb513da126
Author: Markos Chandras <mchandras@suse.de>
Date:   Tue Oct 18 21:02:10 2016 +0100

elements: runtime-ssh-host-keys: Add openssh-client mapping for SUSE
    
    The SUSE 'openssh' package contains the openssh client.
    
    Change-Id: Ic1da63b6c62158b128d44ac48a0657d5d7c53f67

commit 18a664dd32a59b03f7869e117688f158a63d534a
Author: Paul Belanger <pabelanger@redhat.com>
Date:   Wed Sep 28 16:03:52 2016 -0400

Don't use ssh-keygen -A for init scripts
    
    We are running into race conditions with glean, which ssh-keygen -A is
    not handling properly.  So, create a new script to first check if the
    file exists, then use 'yes' to disable overwriting of existing files.
    
    Change-Id: Ie82e1e3f832fcc8f32c7e1335c5f0ee16d36f9a8
    Signed-off-by: Paul Belanger <pabelanger@redhat.com>

commit 89c708ce61999501c564e1a629962940c3b5c791
Author: Markos Chandras <mchandras@suse.de>
Date:   Tue Oct 18 19:02:41 2016 +0100

elements: simple-init: Remove SUSE interfaces
    
    Make sure SUSE interfaces are removed as well.
    
    Change-Id: If993dc606217f2ec243392ac2fa588ebae1cce86

commit fdffa56ff2034f943b13996815a9e216c724e3b5
Author: Gregory Haynes <greg@greghaynes.net>
Date:   Wed Aug 17 16:20:36 2016 +0000

Add element for setting sysctl values
    
    Theres a pretty standard workflow for setting a sysctl value which will
    be applied on image boot which was written by tripleo. Lets move this in
    tree as other folks (like Octavia) would like to depend on it.
    
    Change-Id: I3c266870d417cdba3196f5fa65c4cd634ab13173

commit fd5fbdd4b5c15c613133c5a18c8e83b486ec1dd0
Author: Gregory Haynes <greg@greghaynes.net>
Date:   Sat Jun 25 04:20:04 2016 +0000

Make xenial the ubuntu default
    
    This is the new LTS.
    
    Change-Id: I42a6c0520dde8be21df396e7c2e6fb5ae55d2025

tags:

added: in-feature-v2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-24: Fix proposed to diskimage-builder (feature/v2)

#5

Fix proposed to branch: feature/v2
Review: https://review.openstack.org/438010

Revision history for this message

Andreas Florath (ansreas) wrote on 2017-02-24:

#6

The patch in [1] introduces a bug: only the first package specified with '-p' is picked up. There is another (proposed) patch that fixes this patch [2].

[1] https://review.openstack.org/#/c/396702/
[2] https://review.openstack.org/#/c/438010/

Changed in diskimage-builder:
assignee:	Oliver Walsh (owalsh) → Andreas Florath (ansreas)

Revision history for this message

Oliver Walsh (owalsh) wrote on 2017-02-27:

#7

The patch in [1] works. INSTALL_PACKAGES is a bash array, not a space separated list. E.g if you look at the usage the array is expanded https://github.com/openstack/diskimage-builder/blob/111cb510550ab05a0bc760e2075fc1eef9d8bb15/lib/img-functions#L151

Revision history for this message

Andreas Florath (ansreas) wrote on 2017-02-28:

#8

Sorry for the confusion: I missed the array. Everything is fine!

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-28: Change abandoned on diskimage-builder (feature/v2)

#9

Change abandoned by Andreas Florath (<email address hidden>) on branch: feature/v2
Review: https://review.openstack.org/438010
Reason: Sorry for the confusion: I misses the array - everything is fine!

diskimage-builder

support multiple -p options in disk-image-create

Bug Description

Other bug subscribers

Remote bug watches