long delay (40 seconds) in boot due to snapd.seeded.service

[Scott Moser]

This is a default container (bionic) launched from a ppc64el host

inside I see:

$ lxc launch ubuntu-daily:bionic sstest-616
$ lxc exec sstest-616 /bin/bash
root@sstest-616:~# systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @46.675s
└─multi-user.target @46.673s
  └─snapd.seeded.service @6.561s +40.111s
    └─snapd.service @6.448s +102ms
      └─basic.target @6.424s
        └─sockets.target @6.424s
          └─snapd.socket @6.422s +2ms
            └─sysinit.target @6.411s
              └─cloud-init.service @5.378s +1.030s
                └─systemd-networkd-wait-online.service @1.199s +4.175s
                  └─systemd-networkd.service @1.146s +49ms
                    └─network-pre.target @1.144s
                      └─cloud-init-local.service @156ms +987ms
                        └─systemd-journald.socket @136ms
                          └─system.slice @135ms
                            └─-.slice @132ms

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: snapd 2.32.9+18.04
ProcVersionSignature: Ubuntu 4.15.0-24.26-generic 4.15.18
Uname: Linux 4.15.0-24-generic ppc64le
NonfreeKernelModules: veth nbd openvswitch nsh xt_conntrack ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_CHECKSUM xt_comment xt_tcpudp devlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables iptable_mangle bridge stp llc ip6table_filter ip6_tables iptable_filter kvm_hv kvm binfmt_misc zfs zunicode zavl icp zcommon znvpair spl leds_powernv ipmi_powernv ipmi_devintf ibmpowernv uio_pdrv_genirq powernv_op_panel ipmi_msghandler uio powernv_rng vmx_crypto sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 dm_service_time ses enclosure scsi_transport_sas btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_vpmsum crc32c_vpmsum tg3 ipr scsi_dh_emc scsi_dh_rdac scsi_dh_alua dm_multipath
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: ppc64el
Date: Thu Jul 19 17:30:10 2018
ProcEnviron:
 TERM=screen-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
ProcLoadAvg: 0.09 0.61 0.72 1/2304 728
ProcLocks:
 6: POSIX ADVISORY WRITE 219 00:60:153 0 EOF
 8: FLOCK ADVISORY WRITE 212 00:60:142 0 EOF
ProcSwaps:
 Filename Type Size Used Priority
 none virtual 8388544 8388544 0
ProcVersion: Linux version 4.15.0-24-generic (buildd@bos02-ppc64el-005) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3)) #26-Ubuntu SMP Wed Jun 13 08:43:33 UTC 2018
SourcePackage: snapd
UpgradeStatus: No upgrade log present (probably fresh install)
VarLogDump_list: total 0
cpu_cores: Number of cores present = 20
cpu_coreson: Number of cores online = 20
cpu_smt: SMT=8

[Scott Moser (smoser)]

possibly some relevant info, from journalctl -o short-monotonic

[ 540.929982] sstest-616 systemd[1]: snapd.seeded.service: Failed to reset devices.list: Operation not permitted
[ 540.930083] sstest-616 sshd[257]: Server listening on 0.0.0.0 port 22.
[ 540.931309] sstest-616 sshd[257]: Server listening on :: port 22.
[ 540.931397] sstest-616 systemd[1]: Starting Wait until snapd is fully seeded...
<snip 40 second delay>
[ 565.816128] sstest-616 systemd-networkd[155]: Could not set hostname: Method call timed out
[ 580.923210] sstest-616 snapd[211]: 2018/07/19 17:27:06.134123 stateengine.go:101: state ensure error: Get https://api.snapcraft.io/api/v1/snaps/sections: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
[ 581.042394] sstest-616 systemd[1]: Started Wait until snapd is fully seeded.

[Christian Ehrhardt  (paelzer)]

Was maybe the proxy env missing (or ignored) that is neede don Diamond?
Still a bug to block the crit chain for 40s if that is the case, but maybe a thought to check.

[Scott Moser (smoser)]

Right.
So we think this was actually just lack of external connection from the system.
This is still an issue though. There is no requirement on internet connectivity for running a container ('lxc launch ubuntu-daily:bionic').

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.