disk-image-create fails in ovb jobs intermittently

[wes hayutin]

https://logs.rdoproject.org/86/638286/3/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/a7e1c2f/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz#_2019-02-22_19_16_55

19-02-22 19:16:54 | 2019-02-22 19:16:54.312 | + '[' erase == install ']'
2019-02-22 19:16:54 | 2019-02-22 19:16:54.472 | Operation failed: No such file or directory
2019-02-22 19:16:55 | 2019-02-22 19:16:55.169 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
2019-02-22 19:16:55 | 2019-02-22 19:16:55.409 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/tmp/yum
2019-02-22 19:16:55 | 2019-02-22 19:16:55.462 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/tmp/in_target.d
2019-02-22 19:16:55 | 2019-02-22 19:16:55.503 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/sys
2019-02-22 19:16:55 | 2019-02-22 19:16:55.545 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/proc
2019-02-22 19:16:55 | 2019-02-22 19:16:55.582 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/dev/pts
2019-02-22 19:16:55 | 2019-02-22 19:16:55.625 | Unmount /tmp/dib_build.yeD6p3NZ/mnt/dev
2019-02-22 19:16:57 | Exception occured while running the command
2019-02-22 19:16:57 | Traceback (most recent call last):
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 29, in run
2019-02-22 19:16:57 | super(Command, self).run(parsed_args)
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
2019-02-22 19:16:57 | return super(Command, self).run(parsed_args)
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
2019-02-22 19:16:57 | return_code = self.take_action(parsed_args) or 0
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 94, in take_action
2019-02-22 19:16:57 | manager.build()
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/build.py", line 85, in build
2019-02-22 19:16:57 | elements, options, packages, extra_options)
2019-02-22 19:16:57 | File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_builder.py", line 139, in build_image
2019-02-22 19:16:57 | raise subprocess.CalledProcessError(process.returncode, cmd)
2019-02-22 19:16:57 | CalledProcessError: Command '['disk-image-create', '-a', 'amd64', '-o', './ironic-python-agent', '-t', 'qcow2', '-p', 'yum-plugin-priorities,python-hardware-detect', '--min-tmpfs=7', 'centos7', 'selinux-permissive', 'ironic-agent', 'ironic-agent-multipath', 'dynamic-login', 'element-manifest', 'network-gateway', 'enable-packages-install', 'override-pip-and-virtualenv']' returned non-zero exit status 1
2019-02-22 19:16:57 | Command '['disk-image-create', '-a', 'amd64', '-o', './ironic-python-agent', '-t', 'qcow2', '-p', 'yum-plugin-priorities,python-hardware-detect', '--min-tmpfs=7', 'centos7', 'selinux-permissive', 'ironic-agent', 'ironic-agent-multipath', 'dynamic-login', 'element-manifest', 'network-gateway', 'enable-packages-install', 'override-pip-and-virtualenv']' returned non-zero exit status 1

[wes hayutin (weshayutin)]

recreated outside of rdo-cloud on a libvirt node..

2019-02-26 22:07:39.345 | Complete!
2019-02-26 22:07:39.368 | + '[' erase == install ']'
2019-02-26 22:07:39.431 | Operation failed: No such file or directory
2019-02-26 22:07:39.909 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here
2019-02-26 22:07:39.979 | Unmount /tmp/dib_build.jsABHu49/mnt/tmp/yum
2019-02-26 22:07:40.007 | Unmount /tmp/dib_build.jsABHu49/mnt/tmp/in_target.d
2019-02-26 22:07:40.028 | Unmount /tmp/dib_build.jsABHu49/mnt/sys
2019-02-26 22:07:40.055 | Unmount /tmp/dib_build.jsABHu49/mnt/proc
2019-02-26 22:07:40.077 | Unmount /tmp/dib_build.jsABHu49/mnt/dev/pts
2019-02-26 22:07:40.101 | Unmount /tmp/dib_build.jsABHu49/mnt/dev
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 29, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 94, in take_action
    manager.build()
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/build.py", line 85, in build
    elements, options, packages, extra_options)
  File "/usr/lib/python2.7/site-packages/tripleo_common/image/image_builder.py", line 139, in build_image
    raise subprocess.CalledProcessError(process.returncode, cmd)
CalledProcessError: Command '['disk-image-create', '-a', 'amd64', '-o', './ironic-python-agent', '-t', 'qcow2', '-p', 'yum-plugin-priorities,python-hardware-detect', '--min-tmpfs=7', 'centos7', 'selinux-permissive', 'ironic-agent', 'ironic-agent-multipath', 'dynamic-login', 'element-manifest', 'network-gateway', 'enable-packages-install', 'override-pip-and-virtualenv']' returned non-zero exit status 1
Command '['disk-image-create', '-a', 'amd64', '-o', './ironic-python-agent', '-t', 'qcow2', '-p', 'yum-plugin-priorities,python-hardware-detect', '--min-tmpfs=7', 'centos7', 'selinux-permissive', 'ironic-agent', 'ironic-agent-multipath', 'dynamic-login', 'element-manifest', 'network-gateway', 'enable-packages-install', 'override-pip-and-virtualenv']' returned non-zero exit status 1
[zuul@standalone ~]$

[wes hayutin (weshayutin)]

testing w/

export DIB_YUM_REPO_CONF=""
export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF $(ls /etc/yum.repos.d/delorean*)"
export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF $(ls /etc/yum.repos.d/CentOS-Ceph-*)"
export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF $(ls /etc/yum.repos.d/centos-*)"
export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF $(ls /etc/yum.repos.d/quickstart-*)"
export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF $(ls /home/zuul/web-gating.repo)"
export DISTRO_NAME=centos7
export DIB_RELEASE=GenericCloud
export YUM=yum
export DIB_PYTHON=python2
export TRIPLEO_COMMON_PATH="/usr/share/tripleo-common"
export ELEMENTS="/usr/share/tripleo-puppet-elements:"
export ELEMENTS+="/usr/share/tripleo-image-elements:"
export ELEMENTS+="/usr/share/heat-agents"
export ELEMENTS_PATH=$ELEMENTS

disk-image-create -a amd64 -o ./ironic-python-agent -t qcow2 -p yum-plugin-priorities python-hardware-detect --min-tmpfs=7 centos7 selinux-permissive ironic-agent ironic-agent-multipath dynamic-login element-manifest network-gateway enable-packages-install override-pip-and-virtualenv

[Alex Schultz (alex-schultz)]

2019-02-26 22:59:24.746 | Complete!
2019-02-26 22:59:24.771 | + '[' erase == install ']'
2019-02-26 22:59:24.772 | + rm -rf /tmp/ironic-python-agent
2019-02-26 22:59:24.773 | + case "$DIB_INIT_SYSTEM" in
2019-02-26 22:59:24.773 | ++ systemctl --no-pager list-unit-files iptables
2019-02-26 22:59:24.917 | + [[ UNIT FILE STATE
2019-02-26 22:59:24.917 |
2019-02-26 22:59:24.917 | 0 unit files listed. =~ enabled ]]
2019-02-26 22:59:24.917 | ++ svc-map ironic-python-agent
2019-02-26 22:59:24.969 | + systemctl enable openstack-ironic-python-agent.service
2019-02-26 22:59:24.971 | Operation failed: No such file or directory
2019-02-26 22:59:24.978 | ++ diskimage_builder/lib/img-functions:run_in_target:59 : check_break after-error run_in_target bash
2019-02-26 22:59:24.982 | ++ diskimage_builder/lib/common-functions:check_break:135 : echo ''
2019-02-26 22:59:24.984 | ++ diskimage_builder/lib/common-functions:check_break:135 : egrep -e '(,|^)after-error(,|$)' -q
2019-02-26 22:59:24.989 | + diskimage_builder/lib/img-functions:run_in_target:1 : trap_cleanup
2019-02-26 22:59:24.992 | + diskimage_builder/lib/img-functions:trap_cleanup:36 : exitval=1
2019-02-26 22:59:24.995 | + diskimage_builder/lib/img-functions:trap_cleanup:37 : cleanup
2019-02-26 22:59:24.998 | + diskimage_builder/lib/img-functions:cleanup:42 : dib-block-device umount
2019-02-26 22:59:26.638 | DEBUG diskimage_builder.block_device.blockdevice [-] Creating BlockDevice object __init__ /usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py:228
2019-02-26 22:59:26.638 | DEBUG diskimage_builder.block_device.blockdevice [-] Params [{'root-fs-type': 'ext4', 'image-dir': '/tmp/dib_image.PVRNliqY', 'root-label': None, 'build-dir': '/tmp/dib_build.TbaxJNCg', 'config': '/tmp/dib_build.TbaxJNCg/block-device/config.yaml', 'mount-base': '/tmp/dib_build.TbaxJNCg/mnt'}] __init__ /usr/lib/python2.7/site-packages/diskimage_builder/block_device/blockdevice.py:231
2019-02-26 22:59:26.642 | INFO diskimage_builder.block_device.blockdevice [-] State already cleaned - no way to do anything here

[Alex Schultz (alex-schultz)]

2019-02-26 23:26:37.387 | access("/etc/systemd/system/openstack-ironic-python-agent.service", F_OK) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | access("/run/systemd/system/openstack-ironic-python-agent.service", F_OK) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | access("/usr/local/lib/systemd/system/openstack-ironic-python-agent.service", F_OK) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | access("/usr/lib/systemd/system/openstack-ironic-python-agent.service", F_OK) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | access("/etc/rc.d/init.d/openstack-ironic-python-agent", F_OK) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | lstat("/etc", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/etc/systemd", {st_mode=S_IFDIR|0755, st_size=151, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/etc/systemd/system", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/etc", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/etc/systemd", {st_mode=S_IFDIR|0755, st_size=151, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/etc/systemd/system", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/run", {st_mode=S_IFDIR|0755, st_size=221, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/run/systemd", {st_mode=S_IFDIR|0755, st_size=113, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/run/systemd/system", 0x7ffd6d57dc90) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | lstat("/usr", {st_mode=S_IFDIR|0755, st_size=155, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/local", {st_mode=S_IFDIR|0755, st_size=131, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/local/lib", {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/local/lib/systemd", 0x7ffd6d57dc90) = -1 ENOENT (No such file or directory)
2019-02-26 23:26:37.387 | lstat("/usr", {st_mode=S_IFDIR|0755, st_size=155, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/lib", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/lib/systemd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/lib/systemd/system", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr", {st_mode=S_IFDIR|0755, st_size=155, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/lib", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
2019-02-26 23:26:37.387 | lstat("/usr/lib/systemd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/usr/lib/systemd/system", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/etc", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/etc/rc.d", {st_mode=S_IFDIR|0755, st_size=127, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/etc/rc.d/init.d", {st_mode=S_IFDIR|0755, st_size=70, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/etc", {st_mode=S_IFDIR|0755, st_size=8192, ...}) = 0
2019-02-26 23:26:37.388 | lstat("/etc/rc.d", {st_mode=S_IFDIR|0755, st_size=127, ...}) = 0
2019-02-26 23:26:37.388 | gettid() ...

[Alex Schultz (alex-schultz)]

We need https://review.openstack.org/#/c/637162/. By removing git, we're uninstalling the openstack-ironic-python-agent so it's failing when it tries to enable the service

2019-02-26 23:23:41.621 | Removed:
2019-02-26 23:23:41.621 | cloud-init.x86_64 0:18.2-1.el7.centos.1 git.x86_64 0:1.8.3.1-20.el7
2019-02-26 23:23:41.621 |
2019-02-26 23:23:41.621 | Dependency Removed:
2019-02-26 23:23:41.621 | openstack-ironic-python-agent.noarch 0:3.5.1-0.20190209002413.c5749cc.el7
2019-02-26 23:23:41.621 | perl-Git.noarch 0:1.8.3.1-20.el7
2019-02-26 23:23:41.621 | python-testtools.noarch 0:1.8.0-2.el7
2019-02-26 23:23:41.621 | python2-debtcollector.noarch 0:1.20.0-0.20190109185253.d3165af.el7
2019-02-26 23:23:41.622 | python2-fixtures.noarch 0:3.0.0-7.el7
2019-02-26 23:23:41.622 | python2-hardware-detect.noarch 0:0.18.1-0.20190212114100.255924e.el7
2019-02-26 23:23:41.622 | python2-ironic-lib.noarch 0:2.16.1-0.20190121175330.653b59a.el7
2019-02-26 23:23:41.622 | python2-ironic-python-agent.noarch 0:3.5.1-0.20190209002413.c5749cc.el7
2019-02-26 23:23:41.622 | python2-oslo-concurrency.noarch 0:3.29.0-0.20190115144635.0767ddf.el7
2019-02-26 23:23:41.622 | python2-oslo-config.noarch 2:6.8.0-0.20190124103752.8e17ded.el7
2019-02-26 23:23:41.622 | python2-oslo-context.noarch 0:2.22.0-0.20190114133604.f65408d.el7
2019-02-26 23:23:41.622 | python2-oslo-i18n.noarch 0:3.23.0-0.20190111180822.a5fde9a.el7
2019-02-26 23:23:41.622 | python2-oslo-log.noarch 0:3.42.2-0.20190118111346.1babd44.el7
2019-02-26 23:23:41.622 | python2-oslo-serialization.noarch 0:2.28.1-0.20190115052822.0371c1d.el7
2019-02-26 23:23:41.622 | python2-oslo-service.noarch 0:1.36.0-0.20190201143914.3c645a7.el7
2019-02-26 23:23:41.622 | python2-oslo-utils.noarch 0:3.40.2-0.20190204050744.4eb6194.el7
2019-02-26 23:23:41.622 | python2-pbr.noarch 0:5.1.2-2.el7
2019-02-26 23:23:41.622 | python2-stevedore.noarch 0:1.30.0-0.20190111130314.5b39123.el7