inject_password fails with python3

[Dr. Jens Harbott]

Originally reported in #openstack-nova:

14:44 < lvdombrkr> hello guys, trying to inject admin_password (inject_password=true ) into image but when creating instance get this error in nova-compute.log
14:45 < lvdombrkr> 2020-06-06 14:53:50.188 6 WARNING nova.virt.disk.api [req-94f485ca-944c-40e9-bf14-c8b8dbe09a7b 052d02306e6746a4a3e7e5449de49f8c
                   413a4cadf9734fca9ec3e5e6192a446f - default default] Ignoring error injecting admin_password into image (a bytes-like object is required, not 'str')
14:45 < lvdombrkr> Train + Centos8

Can reproduce on master on devstack by installing python3-guestfs and setting

[libvirt]
inject_partition = -1
inject_password = true

in nova-cpu.conf. Backtrace after adding a hard "raise" into inject_data_into_fs():

Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.virt.libvirt.driver [None req-47214a25-b56a-4135-83bb-7c5ff4c86ca6 demo demo] [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Error injecting data into image 4b3e63a6-b3c4-4de5-b515-cc286e7d5c48 (a bytes-like object is required, not 'str')
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [None req-47214a25-b56a-4135-83bb-7c5ff4c86ca6 demo demo] [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Instance failed to spawn: TypeError: a bytes-like object is required, not 'str'
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] Traceback (most recent call last):
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/compute/manager.py", line 2614, in _build_resources
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] yield resources
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/compute/manager.py", line 2374, in _build_and_run_instance
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self.driver.spawn(context, instance, image_meta,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3604, in spawn
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] created_instance_dir, created_disks = self._create_image(
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3991, in _create_image
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] created_disks = self._create_and_inject_local_root(
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4119, in _create_and_inject_local_root
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self._inject_data(backend, instance, injection_info)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3894, in _inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] LOG.error('Error injecting data into image '
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] self.force_reraise()
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] six.reraise(self.type_, self.value, self.tb)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] raise value
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3887, in _inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] disk_api.inject_data(disk.get_model(self._conn),
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 368, in inject_data
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] return inject_data_into_fs(fs, key, net, metadata, admin_password,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 461, in inject_data_into_fs
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] inject_func(inject_val, fs)
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 597, in _inject_admin_password_into_fs
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] new_shadow_data = _set_passwd(admin_user, admin_passwd,
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] File "/opt/stack/nova/nova/virt/disk/api.py", line 644, in _set_passwd
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] p_file = passwd_data.split("\n")
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863] TypeError: a bytes-like object is required, not 'str'
Jun 06 15:48:39 jh-devstack-focal-01a nova-compute[2983293]: ERROR nova.compute.manager [instance: 5604d60c-61c9-49b5-8786-ff5144817863]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/733982

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jens Harbott (frickler) (<email address hidden>) on branch: master
Review: https://review.opendev.org/733982
Reason: This is wrong, the issue happens only with guestfs, localfs seems to works with str properly. This exceeds my current capacity, so I'm abandoning to let others take over.

[Nathaniel Sherry (nsherry4)]

The issue seems to be that guestfs under Python 3 returns bytes rather than str for file reads while still accepting str for replacing and appending. This patch modifies the guestfs backend's read_file method to return str data for Python 3 while preserving existing behaviour for Python 2. I suppose as long as all the files being read are text, this should be fine.

[Martin Kopec (mkopec)]

We disabled file injection tests on tempest-all job due to this bug:
https://review.opendev.org/c/openstack/tempest/+/774701

[Lee Yarwood (lyarwood)]

https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f caused this, looking at a fix now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/787712

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/787712
Committed: https://opendev.org/openstack/nova/commit/606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e
Submitter: "Zuul (22348)"
Branch: master

commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

    guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

    libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
    of read_file from string to bytes.

    https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    As we don't check the version of libguestfs installed this change
    handles both the original behaviour where a string is returned and the
    newer behaviour by decoding any returned bytes to a string.

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/787839

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/787901

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/787902

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/787903

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/787839
Committed: https://opendev.org/openstack/nova/commit/e98935f705a931e3ae666820387a419be1622b64
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit e98935f705a931e3ae666820387a419be1622b64
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

    guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

    libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
    of read_file from string to bytes.

    https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    As we don't check the version of libguestfs installed this change
    handles both the original behaviour where a string is returned and the
    newer behaviour by decoding any returned bytes to a string.

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
    (cherry picked from commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/787901
Committed: https://opendev.org/openstack/nova/commit/210abc09b857880aa7665fb540f46cb53662a600
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 210abc09b857880aa7665fb540f46cb53662a600
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

    guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

    libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
    of read_file from string to bytes.

    https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    As we don't check the version of libguestfs installed this change
    handles both the original behaviour where a string is returned and the
    newer behaviour by decoding any returned bytes to a string.

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
    (cherry picked from commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e)
    (cherry picked from commit e98935f705a931e3ae666820387a419be1622b64)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 23.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 22.2.1 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/787902
Committed: https://opendev.org/openstack/nova/commit/637402eec7f3f7f552bfc30d16c5960e3a651395
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 637402eec7f3f7f552bfc30d16c5960e3a651395
Author: Lee Yarwood <email address hidden>
Date: Fri Apr 23 12:14:42 2021 +0100

    guestfs: With libguestfs >= v1.41.1 decode returned bytes to string

    libguestfs >= v1.41.1 and commit 0ee02e0117527 changed the return type
    of read_file from string to bytes.

    https://github.com/libguestfs/libguestfs/commit/0ee02e0117527b86a31b2a88a14994ce7f15571f

    As we don't check the version of libguestfs installed this change
    handles both the original behaviour where a string is returned and the
    newer behaviour by decoding any returned bytes to a string.

    Closes-Bug: #1882421
    Change-Id: I1c12b2903c1e5bf3a88394493456ad33233f3cd8
    (cherry picked from commit 606d588e3eca1d88ad26b4c2cfa3f2e1d5ed553e)
    (cherry picked from commit e98935f705a931e3ae666820387a419be1622b64)
    (cherry picked from commit 210abc09b857880aa7665fb540f46cb53662a600)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 21.2.3 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/787903
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.