(lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3

Bug #1840068 reported by Matt Riedemann on 2019-08-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Matt Riedemann
Queens
Medium
sean mooney
Rocky
Medium
sean mooney
Stein
Medium
sean mooney

Bug Description

Seen in the nova-lxc CI job here:

https://logs.opendev.org/24/676024/2/experimental/nova-lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911

Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last):
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] yield resources
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info=block_device_info)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure=True)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] self.force_reraise()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] six.reraise(self.type_, self.value, self.tb)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] raise value
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5789, in _create_domain_and_network
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info):
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return next(self.gen)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5701, in _lxc_disk_handler
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5651, in _create_domain_setup_lxc
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] container_dir=container_dir)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/api.py", line 396, in setup_container
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] dev = img.mount()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/api.py", line 307, in mount
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] if mounter.do_mount():
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 276, in do_mount
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] status = self.get_dev() and self.map_dev() and self.mnt_dev()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 121, in get_dev
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return self._get_dev_retry_helper()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/api.py", line 168, in _get_dev_retry_helper
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] device = self._inner_get_dev()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return f(*args, **kwargs)
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 74, in _inner_get_dev
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] device = self._allocate_nbd()
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/disk/mount/nbd.py", line 64, in _allocate_nbd
Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] random.shuffle(devices)
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/lib/python3.6/random.py", line 274, in shuffle
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] for i in reversed(range(1, len(x))):
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] TypeError: object of type 'filter' has no len()
Aug 12 23:31:05.048116 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]

Changed in nova:
assignee: nobody → sean mooney (sean-k-mooney)
status: Confirmed → In Progress
Matt Riedemann (mriedem) wrote :

This filter code goes back to 2012 so we could backport the fix further (to pike and ocata) but no one is really using the libvirt+lxc code as far as I can tell, at least not with python3, so we can just backport to the non-extended-maintenance branches unless someone wants to backport them upstream to pike and ocata.

summary: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has
- no len()
+ no len() - python3
Changed in nova:
importance: High → Medium
Changed in nova:
assignee: sean mooney (sean-k-mooney) → Matt Riedemann (mriedem)

Reviewed: https://review.opendev.org/676263
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac
Submitter: Zuul
Branch: master

commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.opendev.org/676496
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e135afec851e33148644d024a9d78e56f962efd4
Submitter: Zuul
Branch: stable/stein

commit e135afec851e33148644d024a9d78e56f962efd4
Author: Sean Mooney <email address hidden>
Date: Tue Aug 13 18:58:41 2019 +0100

    lxc: make use of filter python3 compatible

    _detect_nbd_devices uses the filter
    builtin internally to filter valid devices.

    In python 2, filter returns a list. In python 3,
    filter returns an iterable or generator function.
    This change eagerly converts the result of calling filter
    to a list to preserve the python 2 behaviour under python 3.

    Closes-Bug: #1840068

    Change-Id: I25616c5761ea625a15d725777ae58175651558f8
    (cherry picked from commit fc9fb383c16ecb98b1b546f21e7fabb5f00a42ac)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers