Ubuntu Core provides no console login prompt if network is unavailable

[Daniel van Vugt]

Ubuntu Core provides no console login prompt if network is unavailable.

If I plug ethernet in, it boots to the login prompt fine.

[Daniel van Vugt (vanvugt)]

Hmm, probably not a bug. I just need to find out how to enable getty for systemd.

[Alexander Sack (asac)]

slangasek was the last to touch this... I think you might just find your login prompt on a separate console (try ctrl+F2 etc.).

[Daniel van Vugt (vanvugt)]

Nope, it's not on any VT.

The login prompt doesn't appear (roughly) till DHCP completes. So if you've got no network, you won't get a login prompt.

[Sergio Schvezov (sergiusens)]

in a different bug you mentioned disabling systemd from grub, is that still the case here?

[Daniel van Vugt (vanvugt)]

Ignore my (now hidden) comments about disabling systemd and switching back to init/upstart. That was just an ugly workaround before I figured out what the real problem was.

Obviously we need a solution that works with systemd. Seems like a simple matter of service dependencies/ordering. The getty service is blocked by networking (and DHCP acquiring an IP address?) coming up. It should not be related.

[Steve Langasek (vorlon)]

I'm looking through the systemd ordering now; I've found a couple of issues so far:

 - ifupdown: /lib/systemd/system/network-online.target.wants/ifup-wait-all-auto.service is a broken symlink to ../ifup-all-auto.service, looks like this should be ../ifup-wait-all-auto.service instead.

 - cloud-init: cloud-init.service declares a 'Requires=network.target', but the definition of network.target is:

       network.target
           This unit is supposed to indicate when network functionality is
           available, but it is only very weakly defined what that is supposed
           to mean, with one exception: at shutdown, a unit that is ordered
           after network.target will be stopped before the network -- to
           whatever level it might be set up then -- is shut down. It is hence
           useful when writing service files that require network access on
           shutdown, which should order themselves after this target, but not
           pull it in. Also see Running Services After the Network is up[1]
           for more information. Also see network-online.target described
           above.

           systemd automatically adds dependencies of type After= for this
           target unit to all SysV init script service units with an LSB
           header referring to the "$network" facility.

Since cloud-init is not a provider of network connectivity, it must not use 'Requires=network.target'. It's possible that this should be a dependency on 'network-online.target' instead.

Still looking.

[Steve Langasek (vorlon)]

Sorry, that part of the systemd.special(7) manpage doesn't actually explain the problem with network.target. The problem is described here:

SPECIAL PASSIVE SYSTEM UNITS
       A number of special system targets are defined that can be used to
       properly order boot-up of optional services. These targets are
       generally not part of the initial boot transaction, unless they are
       explicitly pulled in by one of the implementing services. Note
       specifically that these passive target units are generally not pulled
       in by the consumer of a service, but by the provider of the service.
       This means: a consuming service should order itself after these targets
       (as appropriate), but not pull it in. A providing service should order
       itself before these targets (as appropriate) and pull it in (via a
       Wants= type dependency).

network.target is one of these "special passive system units".

Locally fixing these bugs in cloud-init and ifupdown does not unblock the networkless boot.

[Steve Langasek (vorlon)]

The remaining issue is that /etc/init.d/networking is sequenced before network.target, with no timeout.

# cat /run/systemd/generator/networking.service.d/50-insserv.conf-\$network.conf
# Automatically generated by systemd-insserv-generator

[Unit]
Wants=network.target
Before=network.target
#

ifupdown has a systemd hook that's intended to reproduce the "network-failsafe" behavior that we had under upstart; but this only hooks into the network-online target, not the earlier network target which is the one that actually blocks boot (nothing on a snappy system actually references network-online.target).

We should definitely not be blocking network.target indefinitely on boot waiting for the results of /etc/init.d/networking. There needs to be some kind of timeout to allow the system to continue booting when a network device configured in /etc/network can't be brought online - while also allowing the system to continue trying to bring that network device up in the background.

Since /run/systemd/generator/networking.service.d/50-insserv.conf-\$network.conf is created by systemd-insserv-generator, opening a systemd task as well.

[Steve Langasek (vorlon)]

Martin, I'm assigning the ifupdown and systemd tasks to you, as I'm not sure how to fix this. Can you please have a look?

[Martin Pitt (pitti)]

Will have a thorough look after the holidays, a dependency chain like this might explain it?

 getty@.service → systemd-user-sessions.service → remote-fs.target → network-online.target

Incidentally that affects desktops as well, lightdm is now starting only after the network is up. This should be fixed indeed.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ifupdown - 0.7.48.1ubuntu7

---------------
ifupdown (0.7.48.1ubuntu7) vivid; urgency=medium

  * Fix symlink target of ifup-wait-all-auto.service so that it works as
    intended. LP: #1425376.
 -- Steve Langasek <email address hidden> Fri, 03 Apr 2015 17:42:13 -0700

[Martin Pitt (pitti)]

As for what creates the "Before=network.target" for networking.service (aka /etc/init.d/networking): this is defined in /etc/insserv.conf.

> We should definitely not be blocking network.target indefinitely on boot waiting for the results of /etc/init.d/networking.

Agreed. Conceptually we should keep the Before= so that on shutdown networking.service gets stopped after every service which provides something on the network (and thus can shut down cleanly), but on boot it's fine to asynchronously do the ifup in the background.

However, I cannot actually reproduce this on the current devel-proposed image? I booted with "-net nic,model=virtio -net none" to simulate an unconnected ethernet port, and the only "hanging" job that I get is <email address hidden> (expectedly). network.target and getty are up, but networking.service didn't even run. But it turns out this is due to bug 1440978, I'll address this first.

[Martin Pitt (pitti)]

With bug 1440978 fixed locally I can now reproduce this. <email address hidden> indeed takes some 3 minutes to time out, then the getty finally appears. So let's run the ifup@ in the background; at least in Ubuntu we didn't block on these under upstart either, and as I wrote before the main reason for the Before=network.target is to ensure a proper shutdown order.

[Martin Pitt (pitti)]

http://anonscm.debian.org/cgit/pkg-systemd/systemd.git/commit/?h=ubuntu&id=07d6481d7a7a5

[Steve Langasek (vorlon)]

On Tue, Apr 07, 2015 at 06:43:35AM -0000, Martin Pitt wrote:
> So let's run the ifup@ in the background; at least in Ubuntu we didn't
> block on these under upstart either,

I'm not sure what you mean by this. We absolutely did block on our network
interfaces when using upstart; just with a 2 minute timeout for any
misconfigured interfaces.

The code change itself appears reasonable to me.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

[Martin Pitt (pitti)]

Steve Langasek [2015-04-07 14:22 -0000]:
> I'm not sure what you mean by this. We absolutely did block on our network
> interfaces when using upstart; just with a 2 minute timeout for any
> misconfigured interfaces.

Right, but that should only be done by services which depend on
network-online.target IMHO, *not* network.target. The former is done
by ifup-wait-all-auto.service. I. e. we don't need to block services
which merely want network.target on all interfaces to be up.

That's at least the intention. Did I misunderstand you?

[Steve Langasek (vorlon)]

On Tue, Apr 07, 2015 at 02:35:35PM -0000, Martin Pitt wrote:
> Steve Langasek [2015-04-07 14:22 -0000]:
> > I'm not sure what you mean by this. We absolutely did block on our network
> > interfaces when using upstart; just with a 2 minute timeout for any
> > misconfigured interfaces.

> Right, but that should only be done by services which depend on
> network-online.target IMHO, *not* network.target. The former is done
> by ifup-wait-all-auto.service. I. e. we don't need to block services
> which merely want network.target on all interfaces to be up.

> That's at least the intention. Did I misunderstand you?

I agree with the code change. I'm just questioning the "we didn't do this
with upstart" claim - upstart doesn't have any equivalent to
network.target, only network-online.target.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 219-6ubuntu2

---------------
systemd (219-6ubuntu2) vivid; urgency=medium

  * Merge fixes from Debian experimental branch.
  * ifup@.service: Run ifup in the background during boot, to avoid blocking
    network.target. (LP: #1425376)
  * Drop hwdb-update dependency from udev-trigger.service, which got
    introduced in v219-stable. This causes udev and plymouth to start too late
    and isn't really needed in Ubuntu yet as we don't support stateless
    systems yet and handle hwdb.bin updates through dpkg triggers. This can be
    dropped again with initramfs-tools 0.117.

systemd (219-7) UNRELEASED; urgency=medium

  [ Martin Pitt ]
  * Make systemd-sysv's dependency to systemd unversioned. The package just
    contains 6 symlinks and thus isn't sensitive at all against version
    mismatches. This avoids running into circular dependencies when testing
    local debs.
  * Revert "udev: Drop hwdb-update dependency" and replace with upstream patch
    which moves it to systemd-udev-trigger.service.
  * display-managers autopkgtest: Properly wait until all jobs are finished,
    instead of a static sleep.

  [ Adam Conrad ]
  * debian/systemd.{triggers,postinst}: Trigger a systemctl daemon-reload
    when init scripts are installed or removed (Closes: #766429)
 -- Martin Pitt <email address hidden> Tue, 07 Apr 2015 16:38:30 +0200

[Steve Langasek (vorlon)]

Confirmed that the systemd fixes have the correct effect in the latest devel-proposed version. Thanks, Martin! Now we just need the cloud-init changes.

[Steve Langasek (vorlon)]

The last piece of this is cloud-init, which has been uploaded and is awaiting freeze review.

[Steve Langasek (vorlon)]

Ok, don't know how this ended up with the wrong bug number in the changelog, but cloud-init has been uploaded and accepted into vivid.

cloud-init (0.7.7~bzr1088-0ubuntu3) vivid; urgency=medium

  * systemd/*.service: don't declare a Wants/Requires on network.target; this
    is a passive target that should only be pulled in by implementors of the
    networking service. The requirement for network needs to be
    expressed as a dependency on network-online.target. LP: #1440180.

[Mike Carifio (carifio)]

Does the image http://releases.ubuntu.com/15.04/ubuntu-15.04-snappy-amd64-generic.img.xz have this fix? Still can't login after spinning up the image using kvm. Is there a workaround to login based on the kvm directions at https://developer.ubuntu.com/en/snappy/start/? Please advise. Thanks.

[Mike Carifio (carifio)]

Nevermind, it was pilot error on my part. I redid the directions and can both log in directly and ssh via port 8022. Good to see my track record of finding every possible way of failing remains intact...