nova-lxd

Booting with config drive fails rootwrap

Bug #1747720 reported by Michael Johnson on 2018-02-06

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	nova-lxd	Fix Committed	Medium	Unassigned

Bug Description

Booting an instance that uses config drive fails with a nova-rootwrap failure.

Fresh devstack, using nova-lxd plugin, commit 16d06d149ed979745aa0ccd2b6db23122d17b57f

Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [None req-dad055f7-22c2-42fd-ba8e-b3e60a7045de service nova] [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Instance failed to spawn: ProcessExecutionError: Unexpected error while running command
.
Feb 06 09:21:39 novalxd nova-compute[115709]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -o loop,uid=1001,gid=110 /opt/stack/data/nova/instances/instance-00000003/configdrive.iso /tmp/tmpjUu5y0
Feb 06 09:21:39 novalxd nova-compute[115709]: Exit code: 99
Feb 06 09:21:39 novalxd nova-compute[115709]: Stdout: u''
Feb 06 09:21:39 novalxd nova-compute[115709]: Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: mount -o loop,uid=1001,gid=110 /opt/stack/data/nova/instances/instance-00000003/configdrive.iso /tmp/tmpjUu5y0 (no filter matched)\n'
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Traceback (most recent call last):
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova/nova/compute/manager.py", line 2249, in _build_resources
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] yield resources
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova/nova/compute/manager.py", line 2033, in _build_and_run_instance
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] block_device_info=block_device_info)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 568, in spawn
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] network_info)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 1237, in _add_configdrive
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] run_as_root=True)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova/nova/utils.py", line 230, in execute
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] return RootwrapProcessHelper().execute(*cmd, **kwargs)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/opt/stack/nova/nova/utils.py", line 113, in execute
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] return processutils.execute(*cmd, **kwargs)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] cmd=sanitized_cmd)
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] ProcessExecutionError: Unexpected error while running command.
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -o loop,uid=1001,gid=110 /opt/stack/data/nova/instances/instance-00000003/configdrive.iso
/tmp/tmpjUu5y0
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Exit code: 99
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Stdout: u''
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: mount -o loop,uid=1001,gid=110 /opt/stack/data/nova/instances/instance-00000003/config
drive.iso /tmp/tmpjUu5y0 (no filter matched)\n'
Feb 06 09:21:39 novalxd nova-compute[115709]: ERROR nova.compute.manager [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e]
Feb 06 09:21:39 novalxd nova-compute[115709]: INFO nova.compute.manager [None req-dad055f7-22c2-42fd-ba8e-b3e60a7045de service nova] [instance: 1a412095-07e6-4cdd-a1a7-c71c974e738e] Terminating instance

Revision history for this message

Michael Johnson (johnsom) wrote on 2018-02-06:

Looks like the same with umount

Revision history for this message

Alex Kavanagh (ajkavanagh) wrote on 2018-05-15:

Download full text (4.9 KiB)

Yes, I get the same thing:

May 15 10:45:28 devstack nova-compute[3085]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2
May 15 10:45:28 devstack nova-compute[3085]: Exit code: 99
May 15 10:45:28 devstack nova-compute[3085]: Stdout: u''
May 15 10:45:28 devstack nova-compute[3085]: Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2 (no filter matched)\n'
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Traceback (most recent call last):
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova/nova/compute/manager.py", line 2237, in _build_resources
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] yield resources
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova/nova/compute/manager.py", line 2017, in _build_and_run_instance
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] block_device_info=block_device_info)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 569, in spawn
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] network_info)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 1273, in _add_configdrive
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] run_as_root=True)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova/nova/utils.py", line 236, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] return RootwrapProcessHelper().execute(*cmd, **kwargs)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/opt/stack/nova/nova/utils.py", line 119, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] return processutils.execute(*cmd, **kwargs)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459...

Yes, I get the same thing:

May 15 10:45:28 devstack nova-compute[3085]: Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2
May 15 10:45:28 devstack nova-compute[3085]: Exit code: 99
May 15 10:45:28 devstack nova-compute[3085]: Stdout: u''
May 15 10:45:28 devstack nova-compute[3085]: Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2 (no filter matched)\n'
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Traceback (most recent call last):
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova/nova/compute/manager.py", line 2237, in _build_resources
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     yield resources
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova/nova/compute/manager.py", line 2017, in _build_and_run_instance
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     block_device_info=block_device_info)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 569, in spawn
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     network_info)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova-lxd/nova/virt/lxd/driver.py", line 1273, in _add_configdrive
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     run_as_root=True)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova/nova/utils.py", line 236, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     return RootwrapProcessHelper().execute(*cmd, **kwargs)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/opt/stack/nova/nova/utils.py", line 119, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     return processutils.execute(*cmd, **kwargs)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]     cmd=sanitized_cmd)
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] ProcessExecutionError: Unexpected error while running command.
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Exit code: 99
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Stdout: u''
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e] Stderr: u'/usr/local/bin/nova-rootwrap: Unauthorized command: mount -o loop,uid=1000,gid=110 /opt/stack/data/nova/instances/instance-00000001/configdrive.iso /tmp/tmp086sM2 (no filter matched)\n'
May 15 10:45:28 devstack nova-compute[3085]: ERROR nova.compute.manager [instance: e32e065f-b76f-4b0a-8a89-f5e60bf1459e]

As to why it's happening looks to be because the mount and umount commands are missing from:

/etc/nova/rootmap.d/lxd.filters

which currently contains:

root@devstack:/etc/nova/rootwrap.d# cat lxd.filters 
# nova-rootwrap filters for compute nodes running nova-lxd
# This file should be owned by (and only-writable by) the root user

[Filters]
zfs: CommandFilter, zfs, root
zpool: CommandFilter, zpool, root
btrfs: CommandFilter, btrfs, root
chown: CommandFilter, chown, root
chmod: CommandFilter, chmod, root

I think, according to https://docs.openstack.org/nova/queens/admin/root-wrap-reference.html that it probably should include mount and umount.

Changed in nova-lxd:
status:	New → Triaged
importance:	Undecided → Medium

Alex Kavanagh (ajkavanagh) on 2018-05-15

Changed in nova-lxd:
assignee:	nobody → Alex Kavanagh (ajkavanagh)
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-05-16: Fix merged to nova-lxd (master)

Reviewed: https://review.openstack.org/568649
Committed: https://git.openstack.org/cgit/openstack/nova-lxd/commit/?id=bd4378a74fd6678bed121a43609f142569915c9d
Submitter: Zuul
Branch: master

commit bd4378a74fd6678bed121a43609f142569915c9d
Author: Alex Kavanagh <email address hidden>
Date: Tue May 15 18:25:46 2018 +0100

Allow nova-lxd to also use mount & umount in rootwrap

For config drive purposes, nova-lxd needs to call mount and umount.
These need to be added to the rootwrap.d/lxd.filters file.

Change-Id: Id04558cc57c95d09e23f17208c5e275fc9702504
Closes-Bug: #1747720

Changed in nova-lxd:
status:	In Progress → Fix Released

Alex Kavanagh (ajkavanagh) on 2018-05-16

Changed in nova-lxd:
status:	Fix Released → Fix Committed

Alex Kavanagh (ajkavanagh) on 2020-07-23

Changed in nova-lxd:
assignee:	Alex Kavanagh (ajkavanagh) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.