resolvconf not updated by /lib/systemd/system/ifup@.service.d/open-iscsi.conf

[Scott Moser]

I noticed this under MAAS when installing wily.
/etc/resolv.conf (managed by resolvconf) does not get updated by open-iscsi's systemd job.

I can recreate this outside of MAAS with tgt and maas images. tools to help with that are at:
  https://code.launchpad.net/~smoser/maas/maas-ephemeral-sniff

after system booted:
$ cat /run/initramfs/open-iscsi.interface
eth0

$ cat /run/network/ifstate
 lo=lo

$ cat /run/net-eth0.conf
DEVICE='eth0'
PROTO='dhcp'
IPV4ADDR='192.168.122.89'
IPV4BROADCAST='192.168.122.255'
IPV4NETMASK='255.255.255.0'
IPV4GATEWAY='192.168.122.1'
IPV4DNS0='192.168.122.1'
IPV4DNS1='0.0.0.0'
HOSTNAME='maas-enlist'
DNSDOMAIN=''
NISDOMAIN=''
ROOTSERVER='192.168.122.1'
ROOTPATH=''
filename=''
UPTIME='19'
DHCPLEASETIME='3600'
DOMAINSEARCH=''

$ systemctl status --full open-iscsi
● open-iscsi.service - LSB: Starts and stops the iSCSI initiator services and logs in to default targets
   Loaded: loaded (/etc/init.d/open-iscsi)
   Active: active (running) since Tue 2015-06-09 15:08:20 UTC; 12min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 543 ExecStart=/etc/init.d/open-iscsi start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/open-iscsi.service
           ├─573 /usr/sbin/iscsid
           └─574 /usr/sbin/iscsid

Jun 09 15:08:20 ubuntu open-iscsi[543]: ...done.
Jun 09 15:08:20 ubuntu systemd[1]: Child 543 belongs to open-iscsi.service
Jun 09 15:08:20 ubuntu systemd[1]: open-iscsi.service: control process exited, code=exited status=0
Jun 09 15:08:20 ubuntu systemd[1]: open-iscsi.service got final SIGCHLD for state start
Jun 09 15:08:20 ubuntu systemd[1]: open-iscsi.service changed start -> running
Jun 09 15:08:20 ubuntu systemd[1]: Job open-iscsi.service/start finished, result=done
Jun 09 15:08:20 ubuntu systemd[1]: Started LSB: Starts and stops the iSCSI initiator services and logs in to default targets.
Jun 09 15:08:21 ubuntu iscsid[573]: iSCSI daemon with pid=574 started!
Jun 09 15:08:21 ubuntu iscsid[573]: Could not read data from db. Using default and currently negotiated values
Jun 09 15:08:23 ubuntu iscsid[573]: connection1:0 is operational after recovery (1 attempts)

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

$ cat /proc/cmdline
nomodeset iscsi_target_name=tgt-boot-test-S2s2hi iscsi_target_ip=192.168.1.132 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-192.168.1.132:3260-iscsi-tgt-boot-test-S2s2hi-lun-1 overlayroot=tmpfs console=ttyS0 console=tty1 debug ds=nocloud-net;seedfrom=http://192.168.1.132:32600/

Related bugs:
 * bug 1432829: resolv.conf not updated correctly for interfaces configured in initramfs
 * bug 1501033: transient error results in /etc/resolv.conf not populated in iscsi root

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: open-iscsi 2.0.873-3ubuntu11
ProcVersionSignature: User Name 3.19.0-20.20-generic 3.19.8
Uname: Linux 3.19.0-20-generic x86_64
ApportVersion: 2.17.3-0ubuntu4
Architecture: amd64
Date: Tue Jun 9 15:11:06 2015
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)

[Scott Moser (smoser)]

'systemctl status open-iscsi' was of no relevance.

I believe that /lib/systemd/system/ifup@.service.d/open-iscsi.conf is supposed to be called but is not being called.
that calls /lib/open-iscsi/net-interface-handler which handles updating resolvconf

I've added code to /lib/open-iscsi/net-interface-handler like:
  exec >>/run/open-iscsi.out 2>&1
  echo "$(date -R): $*"
  set -x

and the file doesn't even exist, so it appears not being run.

[Martin Pitt (pitti)]

I tried this on a virgin wily cloud image. I installed open-iscsi, added the set -x/exec 2>/run/open-iscsi.log, and the latter does get called. Can you please give me the output of "sudo systemctl status -l ifup@*.service"?

[Scott Moser (smoser)]

Martin,
# systemctl status -l ifup@*.service; echo $?
0

The difference between your scenario and mine is that the NIC has been brougt up in the initramfs and is using iscsi root, so you can't bounce the nic, so ifup doesnt occur.

$ cat /proc/cmdline
nomodeset iscsi_target_name=tgt-boot-test-mT14s1 iscsi_target_ip=192.168.1.132 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-192.168.1.132:3260-iscsi-tgt-boot-test-mT14s1-lun-1 overlayroot=tmpfs console=ttyS0 console=tty1 debug ds=nocloud-net;seedfrom=http://192.168.1.132:32600/

The README at http://bazaar.launchpad.net/~smoser/maas/maas-ephemeral-sniff/view/head:/README.txt shows how to to do this. The tools there handle setting up and tearing down iscsi target.

[Scott Moser (smoser)]

Just for reference, in vivid (working):

$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.122.1

$ uname -r
3.19.0-18-generic

$ cat /proc/cmdline
nomodeset iscsi_target_name=tgt-boot-test-Pv7KUq iscsi_target_ip=192.168.1.132 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist:BOOTIF ro BOOTIF_DEFAULT=eth0 root=/dev/disk/by-path/ip-192.168.1.132:3260-iscsi-tgt-boot-test-Pv7KUq-lun-1 overlayroot=tmpfs console=ttyS0 console=tty1 debug ds=nocloud-net;seedfrom=http://192.168.1.132:32600/

$ cat /run/initramfs/open-iscsi.interface
eth0

$ cat /run/net-eth0.conf
DEVICE='eth0'
PROTO='dhcp'
IPV4ADDR='192.168.122.89'
IPV4BROADCAST='192.168.122.255'
IPV4NETMASK='255.255.255.0'
IPV4GATEWAY='192.168.122.1'
IPV4DNS0='192.168.122.1'
IPV4DNS1='0.0.0.0'
HOSTNAME='maas-enlist'
DNSDOMAIN=''
NISDOMAIN=''
ROOTSERVER='192.168.122.1'
ROOTPATH=''
filename=''
UPTIME='29'
DHCPLEASETIME='3600'
DOMAINSEARCH=''

$ cat /run/network/ifstate
eth0=eth0
lo=lo

$ sudo systemctl status -l ifup@*.service
sudo: unable to resolve host ubuntu
● <email address hidden> - ifup for eth0
   Loaded: loaded (/lib/systemd/system/ifup@.service; static; vendor preset: enabled)
  Drop-In: /lib/systemd/system/ifup@.service.d
           └─open-iscsi.conf
   Active: active (exited) since Wed 2015-06-10 12:56:45 UTC; 3min 52s ago
  Process: 514 ExecStart=/bin/sh -ec ifup --allow=hotplug %I; ifup --allow=auto %I; if ifquery %I >/dev/null; then ifquery --state %I >/dev/null; fi (code=exited, status=0/SUCCESS)
  Process: 455 ExecStartPre=/lib/open-iscsi/net-interface-handler start (code=exited, status=0/SUCCESS)
 Main PID: 514 (code=exited, status=0/SUCCESS)

Jun 10 12:56:45 ubuntu systemd[1]: <email address hidden> changed start-pre -> running
Jun 10 12:56:45 ubuntu systemd[1]: Job <email address hidden>/start finished, result=done
Jun 10 12:56:45 ubuntu systemd[1]: Started ifup for eth0.
Jun 10 12:56:45 ubuntu systemd[514]: Executing: /bin/sh -ec 'ifup --allow=hotplug eth0; ifup --allow=auto eth0; if ifquery eth0 >/dev/null; then ifquery --state eth0 >/dev/null; fi'
Jun 10 12:56:45 ubuntu sh[514]: ifup: interface eth0 already configured
Jun 10 12:56:45 ubuntu sh[514]: ifup: interface eth0 already configured
Jun 10 12:56:45 ubuntu systemd[1]: Child 514 belongs to <email address hidden>
Jun 10 12:56:45 ubuntu systemd[1]: <email address hidden>: main process exited, code=exited, status=0/SUCCESS
Jun 10 12:56:45 ubuntu systemd[1]: <email address hidden> changed running -> exited
Jun 10 12:56:45 ubuntu systemd[1]: <email address hidden>: cgroup is empty

[Martin Pitt (pitti)]

I reproduced such a VM. The reason is that /etc/network/interfaces specifies the interface as "manual", so ifup (and ifup@.service) never gets called on it as /lib/udev/net.agent only starts those on auto and allow-hotplug interfaces.

This worked "by accident" in vivid as we called ifup@.service on all interfaces (regardless whether they were actually configured in /e/n/interfaces and auto/allow-hotplug), but not any more.

I see that under upstart that wasn't hooked into ifupdown, but came from /etc/init/iscsi-network-interface.conf. This is essentially an udev rule, not tied at all to ifupdown, so we could just replace that with an udev rule to do the same under upstart and systemd.

[Martin Pitt (pitti)]

This works again with adding this udev rule:

$ cat /lib/udev/rules.d/70-iscsi-network-interface.rules
SUBSYSTEM=="net", ACTION=="add", RUN+="/lib/open-iscsi/net-interface-handler start"
SUBSYSTEM=="net", ACTION=="remove", RUN+="/lib/open-iscsi/net-interface-handler stop"

In my test I removed /etc/init/iscsi-network-interface.conf and /lib/systemd/system/ifup@.service.d/open-iscsi.conf as they are redundant with hotplugging. They would still be relevant if someone is supposed to ifup/ifdown the iscsi interface and expect net-interface-handler to run; if we want the latter, then we should keep the upstart job/systemd drop-in (net-interface-handler is idempotent, so running it twice doesn't matter), otherwise we can clean them up.

[Scott Moser (smoser)]

Pitti, that change is fine with me. I can't immediately see a need for someone to ifup or ifdown a device and need that hook to run.

The one thing that is important is that the interface found in /run/initramfs/open-iscsi.interface should not be brought down.

/lib/open-iscsi/net-interface-handler hackily accomplishes that via updating state in /run/network/ifstate

[Martin Pitt (pitti)]

> should not be brought down

Ah, then we don't really need to care about ifupdown integration at all. I. e. even if you do call ifdown, we should *not* invoke "net-interface-handler stop", right? Then we can drop the drop-in and upstart job and just use the udev rule. Thanks!

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-iscsi - 2.0.873-3ubuntu12

---------------
open-iscsi (2.0.873-3ubuntu12) wily; urgency=medium

  * Drop debian/ifup@.service.conf again, it ceased to work with net.agent in
    wily and the whole idea of net-interface-handler is to *stop* ifupdown
    from fiddling with the iscsi interface. Replace with udev rules
    (debian/iscsi-network-interface.rules) which achieves this without relying
    on ifupdown or a particular init system. (LP: #1463461)
  * Drop debian/open-iscsi.iscsi-network-interface.upstart as well, obsolete
    now. Also clean it up on upgrades.

 -- Martin Pitt <email address hidden> Thu, 11 Jun 2015 11:58:29 +0200