Ubuntu

Cannot use open-iscsi inside LXC container

Reported by Elizabeth Krumbach Joseph on 2013-09-17
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Wishlist
Unassigned
lxc (Ubuntu)
Wishlist
Unassigned

Bug Description

Trying to use open-iscsi from within an LXC container, but the iscsi netlink socket does not support multiple namespaces, causing: "iscsid: sendmsg: bug? ctrl_fd 6" error and failure.

Command attempted: iscsiadm -m node -p $ip:$port -T $target --login

Results in:

Exit code: 18
Stdout: 'Logging in to [iface: default, target: $target, portal: $ip,$port] (multiple)'
Stderr: 'iscsiadm: got read error (0/0), daemon died?
iscsiadm: Could not login to [iface: default, target: $target, portal: $ip,$port].
iscsiadm: initiator reported error (18 - could not communicate to iscsid)
iscsiadm: Could not log into all portals'

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: lxc 0.9.0-0ubuntu3.4
ProcVersionSignature: Ubuntu 3.8.0-30.44-generic 3.8.13.6
Uname: Linux 3.8.0-30-generic x86_64
ApportVersion: 2.9.2-0ubuntu8.3
Architecture: amd64
Date: Tue Sep 17 14:38:08 2013
InstallationDate: Installed on 2013-01-15 (245 days ago)
InstallationMedia: Xubuntu 12.10 "Quantal Quetzal" - Release amd64 (20121017.1)
MarkForUpload: True
SourcePackage: lxc
UpgradeStatus: Upgraded to raring on 2013-05-16 (124 days ago)

Elizabeth Krumbach Joseph (lyz) wrote :
Serge Hallyn (serge-hallyn) wrote :

Thanks for reporting this bug.

Your example command says '$ip:$port'. Is the iscsid running on the host
or in the container? Is $ip the ip of the host?

If $ip is the host ip and you just want iscsiadm in the guest to talk to iscsid on the
host, that should work.

There are several ways depending on your configuration where netlink sockets
might be being attempted. Could you show strace -f output to show exactly
which fails? (iscsiadm itself should only fail if you're trying offload, which it
doesn't look like you are)

Netlink sockets are per-netns, so if you want to be able to connect to a
netlink socket from another netns, then something will need to open a
socket from the target netns and pass that into the other ns. (This
could be arranged with setns, but only from the host).

Changed in lxc (Ubuntu):
status: New → Incomplete
Clint Byrum (clint-fewbar) wrote :

I can't answer all of the questions, but the basic idea is that an LXC container could mount an iscsi target from inside the container with very little if any cooperation from the host's user space.

I believe other similar systems like nbd use ioctl's to configure such devices, but iscsi uses netlink which I believe is the krux of the problem.

Changed in lxc (Ubuntu):
status: Incomplete → New
Serge Hallyn (serge-hallyn) wrote :

@Clint,

Thanks. Then I see three possible workarounds:

1. The simplest way would be to have iscsid running on the host, and connect to it over tcp from the container.

2. You could also have a container without its own network namespace, and have iscsid running there.

3. You could open the netlink socket from the host network namespace, and pass that into the container.

If none of these suffices, then I'll mark this as affecting the kernel, and it'll take a new kernel feature to make this work. However controlling host devices from a container is in general deemed suboptimal (see user namespaces which may not access many devices at all). To solve the netlink part of the issue we would have to come up with a way to choose which containers may access the netlink socket.

It would still be useful for future consideration of this bug if you could attach an strace of the netlink failure to this bug.

Changed in lxc (Ubuntu):
status: New → Incomplete
Elizabeth Krumbach Joseph (lyz) wrote :

Thanks for your reply, I'll chat with Robert and Clint to see if any of these solutions is reasonable for us.

As a reference point, here's the setup we're using:

Host has 2 VMs: An LXC and an qemu VM

The host has the iscsi_tcp module loaded, which then can be seen and used for the iscsi daemon within the container.

Now, what we're attempting to do is provision the qemu VM via the LXC container using OpenStack's baremetal provisioning tools in a virtualized environment (no nested KVM!), so loosely the procedure is: the LXC container boots the qemu image (we have a nifty power driver) and gives it an address via dnsmasq-dhcp, loads up some things via dnsmasq-tftp (this all works) and then we use iscsi to copy data to the qemu VM. Robert or Clint can chime in with more details (or to clarify/correct!).

Today I ran through the test again and connected to the iscsid daemon inside the container for your strace, attached is the output from: strace -p 1488 -o iscsid_strace_baremetal.txt

Serge Hallyn (serge-hallyn) wrote :

Thanks, that is an interesting strace. What is 192.0.2.69 - is that the host? Could you also start the daemon in the container by hand under strace for a few seconds so we can see exactly how fd 6 is created? (Presumably it is a connection to iscsi_nl_sock, but I'm confused since (a) it managed to get connected and only was refused on send, and (b) if the daemon is talking over tcp then why is it doing netlink at all).

Elizabeth Krumbach Joseph (lyz) wrote :

192.0.2.69 is the IP of the qemu VM that the LXC container is attempting to provision.

I'll load up my test instance soon to get that additional strace.

Elizabeth Krumbach Joseph (lyz) wrote :

Assuming I should use the init script for this, attached output from running the following in the container:

strace -o iscsi-start.txt service open-iscsi start

Serge Hallyn (serge-hallyn) wrote :

Thanks - unfortunately we need the -f flag added to strace to follow forks.

Aha! Attached: strace -f -o iscsi-start_f.txt service open-iscsi start

Changed in lxc (Ubuntu):
status: Incomplete → Confirmed
Changed in lxc (Ubuntu):
importance: Undecided → Wishlist
Changed in linux (Ubuntu):
importance: Undecided → Wishlist

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1226855

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers