using a containerized undercloud "openstack overcloud image upload" fails with can not create regular file

[wes hayutin]

[stack@undercloud ~]$ sudo docker ps -a | grep glance
2f01d3e22aca tripleoupstream/centos-binary-glance-api:latest "kolla_start" 37 minutes ago Up 37 minutes glance_api
fae2e40050fc tripleoupstream/centos-binary-glance-api:latest "kolla_start" 46 minutes ago Exited (0) 45 minutes ago glance_api_db_sync
[stack@undercloud ~]$

cp: cannot create regular file ‘/httpboot/agent.kernel’: No such file or directory
Command 'sudo cp -f "./ironic-python-agent.kernel" "/httpboot/agent.kernel"' returned non-zero exit status 1
Image "overcloud-full-vmlinuz" was uploaded.

Images were uploaded successfully, it looks like the post upload steps failed.
[stack@undercloud ~]$ source stackrc
[stack@undercloud ~]$ glance image-list
+--------------------------------------+------------------------+
| ID | Name |
+--------------------------------------+------------------------+
| 394baeee-8a51-41ba-8ef5-5fd11e505898 | bm-deploy-kernel |
| ec52ff29-7402-4e90-a857-de77ed5af97a | bm-deploy-ramdisk |
| c3a6f128-7bd8-43f2-bbc6-0c7806887973 | overcloud-full |
| 6f345877-ac58-4de2-b847-270f4acee30c | overcloud-full-initrd |
| 61a13874-a622-4ec5-b120-763d4bbaff62 | overcloud-full-vmlinuz |
+--------------------------------------+------------------------+

Trace:

mage "bm-deploy-ramdisk" was uploaded.
+--------------------------------------+-------------------+-------------+-----------+--------+
| ID | Name | Disk Format | Size | Status |
+--------------------------------------+-------------------+-------------+-----------+--------+
| 2225bf53-1c83-456c-8cbb-f835ad10807a | bm-deploy-ramdisk | ari | 355413025 | active |
+--------------------------------------+-------------------+-------------+-----------+--------+
copy agent images to HTTP BOOT dir
cp: cannot create regular file ‘/httpboot/agent.kernel’: No such file or directory
Command 'sudo cp -f "./ironic-python-agent.kernel" "/httpboot/agent.kernel"' returned non-zero exit status 1
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 930, in take_action
    parsed_args.update_existing
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 742, in _file_create_or_update
    self._copy_file(src_file, dest_file)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 703, in _copy_file
    shell=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'sudo cp -f "./ironic-python-agent.kernel" "/httpboot/agent.kernel"' returned non-zero exit status 1
clean_up UploadOvercloudImage: Command 'sudo cp -f "./ironic-python-agent.kernel" "/httpboot/agent.kernel"' returned non-zero exit status 1
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 135, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 180, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 90, in run
    return self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 930, in take_action
    parsed_args.update_existing
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 742, in _file_create_or_update
    self._copy_file(src_file, dest_file)
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_image.py", line 703, in _copy_file
    shell=True)
  File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command 'sudo cp -f "./ironic-python-agent.kernel" "/httpboot/agent.kernel"' returned non-zero exit status 1

END return value: 1

[wes hayutin (weshayutin)]

full trace.

[Bogdan Dobrelya (bogdando)]

This issue seems currently blocks WIP ovb job containing containerized UC: https://review.openstack.org/#/c/542556

[Bogdan Dobrelya (bogdando)]

A few notes:
* --http-boot is picked from $HTTP_BOOT and defaults to /httpboot in the tripleo client
* real file produced with ovb fs001 is lives in /home/zuul/ironic-python-agent.kernel
* we might want to adjust https://review.openstack.org/#/c/543552/ for the containerized UC case, related bug https://bugs.launchpad.net/tripleo/+bug/1748921

[Bogdan Dobrelya (bogdando)]

so we prolly want it copied from ~/ to /var/lib/ironic/httpboot

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/547067

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547067
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=a72604a4411c355f0c0e15e8b54044a55718b639
Submitter: Zuul
Branch: master

commit a72604a4411c355f0c0e15e8b54044a55718b639
Author: Emilien Macchi <email address hidden>
Date: Mon Feb 12 09:13:21 2018 -0800

    Fix overcloud image upload for containerized UC

    Use overcloud image upload --http-boot only for containerized
    undercloud.

    Until we figure out a better way of managing http-boot argument when
    preparing the image, let's add a conditional so we only use it for a
    containerized undercloud and not for the OVB jobs which seems to not
    work as of today.

    Change-Id: I52cdcabc420276d05827e361d4189522fed57176
    Related-Bug: #1748921
    Closes-Bug: #1669186
    Co-authored-by: Emilien Macchi <email address hidden>
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[James Slagle (james-slagle)]

i don't think this is fixed if it's only addressed in quickstart, since it's not specific to CI.

I submitted a patch to update the default value for --http-boot to the right location for the containerized undercloud:

https://review.openstack.org/578490

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573196
Committed: https://git.openstack.org/cgit/openstack/tripleo-docs/commit/?id=d73ca3ad0ddd4f60c0ff379be8c1cde0eac93ea2
Submitter: Zuul
Branch: master

commit d73ca3ad0ddd4f60c0ff379be8c1cde0eac93ea2
Author: Cédric Jeanneret <email address hidden>
Date: Thu Jun 7 13:31:52 2018 +0200

    Added missing information regarding containerized undercloud and ironic

    The doc is lacking the reference to the "--http-boot" option while
    importing images into the undercloud Glance service.

    This commit adds this information, and is based on the following
    merge:
    https://review.openstack.org/#/c/547067/

    Change-Id: I01fa62414b1ddf498026e9a2ae3967b24caa45ba
    Related-Bug: 1669186

[OpenStack Infra (hudson-openstack)]

Change abandoned by James Slagle (<email address hidden>) on branch: master
Review: https://review.openstack.org/578490

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.