(lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3

[Matt Riedemann]

Seen in the nova-lxc CI job here:

https://logs.opendev.org/24/676024/2/experimental/nova-lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911

Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last):
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] yield resources
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info=block_device_info)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure=True)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] self.force_reraise()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] six.reraise(self.type_, self.value, self.tb)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] raise value
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5789, in _create_domain_and_network
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info):
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return next(self.gen)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5701, in _lxc_disk_handler
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5651, in _create_domain_setup_lxc
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] container_dir=container_dir)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/api.py", line 396, in setup_container
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] dev = img.mount()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/api.py", line 307, in mount
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] if mounter.do_mount():
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 276, in do_mount
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] status = self.get_dev() and self.map_dev() and self.mnt_dev()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 121, in get_dev
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return self._get_dev_retry_helper()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 168, in _get_dev_retry_helper
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] device = self._inner_get_dev()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return f(*args, **kwargs)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 74, in _inner_get_dev
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] device = self._allocate_nbd()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 64, in _allocate_nbd
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] random.shuffle(devices)
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/lib/python3.6/random.py", line 274, in shuffle
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] for i in reversed(range(1, len(x))):
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] TypeError: object of type 'filter' has no len()
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]

[Matt Riedemann (mriedem)]

Looks like this is due to nbd mount code not being py3 compliant. We call _detect_nbd_devices which returns a filter:

https://github.com/openstack/nova/blob/e3a55493ba0cb405f14b9090282d93eb230c5715/nova/virt/disk/mount/nbd.py#L44

Which gets passed to random.shuffle:

https://github.com/openstack/nova/blob/e3a55493ba0cb405f14b9090282d93eb230c5715/nova/virt/disk/mount/nbd.py#L64

https://docs.python.org/3.6/library/random.html#random.shuffle

https://docs.python.org/2.7/library/random.html#random.shuffle

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676263

[Matt Riedemann (mriedem)]

This filter code goes back to 2012 so we could backport the fix further (to pike and ocata) but no one is really using the libvirt+lxc code as far as I can tell, at least not with python3, so we can just backport to the non-extended-maintenance branches unless someone wants to backport them upstream to pike and ocata.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676263
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac
Submitter: Zuul
Branch: master

commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/676496

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/676498

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/676500

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/676502

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676496
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e135afec851e33148644d024a9d78e56f962efd4
Submitter: Zuul
Branch: stable/stein

commit e135afec851e33148644d024a9d78e56f962efd4
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8
    (cherry picked from commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676498
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=944c08ff764c1cb598dbebbad8aa51bbdd0a692c
Submitter: Zuul
Branch: stable/rocky

commit 944c08ff764c1cb598dbebbad8aa51bbdd0a692c
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8
    (cherry picked from commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac)
    (cherry picked from commit e135afec851e33148644d024a9d78e56f962efd4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.2.3 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676500
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=04bcb98678c1289810f5a8542b5bf9fe7aeeaa12
Submitter: Zuul
Branch: stable/queens

commit 04bcb98678c1289810f5a8542b5bf9fe7aeeaa12
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8
    (cherry picked from commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac)
    (cherry picked from commit e135afec851e33148644d024a9d78e56f962efd4)
    (cherry picked from commit 944c08ff764c1cb598dbebbad8aa51bbdd0a692c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.13 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676502
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0449fd1f89ab513059188427ff3bc5e9dbbc0546
Submitter: Zuul
Branch: stable/pike

commit 0449fd1f89ab513059188427ff3bc5e9dbbc0546
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    NOTE(mriedem): In this backport the test module needs a mock
    import since change Ib5e585fa4bfb99617cd3ca983674114d323a3cce
    is not in Pike.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8
    (cherry picked from commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac)
    (cherry picked from commit e135afec851e33148644d024a9d78e56f962efd4)
    (cherry picked from commit 944c08ff764c1cb598dbebbad8aa51bbdd0a692c)
    (cherry picked from commit 04bcb98678c1289810f5a8542b5bf9fe7aeeaa12)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova pike-eol release.