azure instances hang starting ephemeral-disk-warning.service

Bug #1626313 reported by Paul Collins
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
walinuxagent (Ubuntu)
New
Undecided
Unassigned

Bug Description

walinuxagent 2.1.3-0ubuntu4.1

I noticed that "juju add-machine" in a xenial environment in azure yields machines that never exit "pending". I logged in and observed:

root 1490 0.0 0.0 91916 29432 ? Ss 22:35 0:00 /usr/bin/python3 /usr/bin/cloud-init modules --mode=config
root 1659 0.0 0.0 4508 748 ? S 22:35 0:00 \_ /bin/sh -c tee -a /var/log/cloud-init-output.log
root 1660 0.0 0.0 4384 684 ? S 22:35 0:00 | \_ tee -a /var/log/cloud-init-output.log
root 2038 0.1 0.0 78136 49656 ? S 22:35 0:01 \_ /usr/bin/apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet dist-upgrade
root 37305 0.0 0.0 19356 5064 pts/0 Ss+ 22:39 0:00 \_ /usr/bin/dpkg --force-confold --force-unsafe-io --status-fd 43 --configure ucf:all libmpdec2:amd64 libpython3.5-stdlib:amd64 python3.5:amd64 libpython3-stdlib:amd64 dh-python:all python3:amd64 libdbus-1-3:amd64 dbus:amd64 libpam-systemd:amd64 libfreetype6:amd64 libfuse2:amd64 grub-common:amd64 grub2-common:amd64 grub-pc-bin:amd64 grub-pc:amd64 udev:amd64 initramfs-tools-bin:amd64 libklibc:amd64 klibc-utils:amd64 initramfs-tools-core:all initramfs-tools:all mdadm:amd64 iproute2:amd64 ifupdown:amd64 ufw:all cron:amd64 fuse:amd64 libdevmapper-event1.02.1:amd64 liblvm2app2.2:amd64 dmsetup:amd64 dmeventd:amd64 liblvm2cmd2.02:amd64 lvm2:amd64 keyboard-configuration:all console-setup-linux:all console-setup:all locales:all libapt-inst2.0:amd64 apt-utils:amd64 libidn11:amd64 libp11-kit0:amd64 libtasn1-6:amd64 libgnutls30:amd64 libsasl2-modules-db:amd64 libsasl2-2:amd64 libldap-2.4-2:amd64 libcurl3-gnutls:amd64 git-man:all git:amd64 curl:amd64 libgnutls-openssl27:amd64 bash-completion:all libglib2.0-0:amd64 libaccountsservice0:amd64 accountsservice:amd64 language-selector-common:all uuid-runtime:amd64 libpython2.7-minimal:amd64 python2.7-minimal:amd64 libpython2.7-stdlib:amd64 python2.7:amd64 libpython2.7:amd64 libtevent0:amd64 libcups2:amd64 libwbclient0:amd64 samba-libs:amd64 python-samba:amd64 samba-common:all samba-common-bin:amd64 dnsmasq-base:amd64 libapparmor-perl:amd64 apparmor:amd64 liblxc1:amd64 lxc-common:amd64 python3-lxc:amd64 lxc1:amd64 lxcfs:amd64 uidmap:amd64 lxd:amd64 open-iscsi:amd64 snap-confine:amd64 ubuntu-core-launcher:amd64 python3-debian:all update-notifier-common:all tzdata:all distro-info-data:all libisc-export160:amd64 libdns-export162:amd64 isc-dhcp-client:amd64 isc-dhcp-common:amd64 less:amd64 lsb-release:all rsyslog:amd64 sudo:amd64 vim-common:amd64 vim-runtime:all libpython3.5:amd64 vim:amd64 vim-tiny:amd64 ubuntu-minimal:amd64 apt-transport-https:amd64 libxml2:amd64 libisc160:amd64 libdns162:amd64 libisccc140:amd64 libisccfg140:amd64 libbind9-140:amd64 liblwres141:amd64 bind9-host:amd64 dnsutils:amd64 command-not-found-data:amd64 python3-commandnotfound:all command-not-found:all dmidecode:amd64 dosfstools:amd64 geoip-database:all hdparm:amd64 irqbalance:amd64 libdrm2:amd64 libglib2.0-data:all libplymouth4:amd64 libsasl2-modules:amd64 lshw:amd64 ltrace:amd64 mtr-tiny:amd64 nano:amd64 openssh-client:amd64 openssh-sftp-server:amd64 openssh-server:amd64 openssl:amd64 plymouth:amd64 plymouth-theme-ubuntu-text:amd64 python3-distupgrade:all python3-update-manager:all ubuntu-release-upgrader-core:all update-manager-core:all python3-gi:amd64 shared-mime-info:amd64 wget:amd64 ubuntu-standard:amd64 xdg-user-dirs:amd64 acpid:amd64 python3-problem-report:all python3-apport:all apport:all byobu:all libeatmydata1:amd64 eatmydata:all ethtool:amd64 libboost-iostreams1.58.0:amd64 libboost-system1.58.0:amd64 libboost-random1.58.0:amd64 libboost-thread1.58.0:amd64 libnspr4:amd64 libnss3-nssdb:all libnss3:amd64 libpam-cgfs:amd64 librados2:amd64 librbd1:amd64 linux-cloud-tools-common:all linux-cloud-tools-4.4.0-38:amd64 linux-cloud-tools-4.4.0-38-generic:amd64 linux-cloud-tools-virtual:amd64 linux-firmware:all linux-headers-4.4.0-38:all linux-headers-4.4.0-38-generic:amd64 linux-image-4.4.0-38-generic:amd64 linux-image-virtual:amd64 linux-headers-generic:amd64 linux-headers-virtual:amd64 linux-virtual:amd64 linux-image-extra-4.4.0-38-generic:amd64 linux-image-generic:amd64 linux-image-extra-virtual:amd64 lxc-templates:amd64 lxc:all lxd-client:amd64 pastebinit:all pollinate:all postfix:amd64 python3-pkg-resources:all python3-cryptography:amd64 python3-software-properties:all software-properties-common:all python3-urllib3:all qemu-block-extra:amd64 qemu-utils:amd64 thermald:amd64 cloud-init:all debootstrap:all open-vm-tools:amd64 walinuxagent:amd64 watershed:amd64
root 58513 0.0 0.0 8632 1696 pts/0 S+ 22:40 0:00 \_ /bin/sh /var/lib/dpkg/info/walinuxagent.postinst configure 2.1.3-0ubuntu4
root 58575 0.0 0.0 18284 4204 pts/0 S+ 22:40 0:00 \_ perl /usr/bin/deb-systemd-invoke start ephemeral-disk-warning.service walinuxagent.service
root 58578 0.0 0.0 25316 1472 pts/0 S+ 22:40 0:00 \_ /bin/systemctl start ephemeral-disk-warning.service

and

ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$ sudo systemctl status ephemeral-disk-warning.service
● ephemeral-disk-warning.service
   Loaded: loaded (/lib/systemd/system/ephemeral-disk-warning.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Sep 21 22:39:33 jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl systemd[1]: Stopped ephemeral-disk-warning.service.
ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$

I noticed that 2.1.3-0ubuntu4.1 says "remove cloud-final.service from both Wants and After", but also note

ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$ grep cloud-final /lib/systemd/system/ephemeral-disk-warning.service
After=cloud-final.service

and in fact:

ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$ last reboot
reboot system boot 4.4.0-15-generic Wed Sep 21 22:35 still running
ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$ grep ' walinuxagent' /var/log/dpkg.log
2016-03-22 16:41:44 install walinuxagent:amd64 <none> 2.1.3-0ubuntu4
2016-03-22 16:41:44 status half-installed walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:41:45 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:41:45 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 configure walinuxagent:amd64 2.1.3-0ubuntu4 <none>
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:06 status half-configured walinuxagent:amd64 2.1.3-0ubuntu4
2016-03-22 16:42:07 status installed walinuxagent:amd64 2.1.3-0ubuntu4
2016-09-21 22:39:33 upgrade walinuxagent:amd64 2.1.3-0ubuntu4 2.1.3-0ubuntu4.1
2016-09-21 22:39:33 status half-configured walinuxagent:amd64 2.1.3-0ubuntu4
2016-09-21 22:39:33 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4
2016-09-21 22:39:33 status half-installed walinuxagent:amd64 2.1.3-0ubuntu4
2016-09-21 22:39:34 status half-installed walinuxagent:amd64 2.1.3-0ubuntu4
2016-09-21 22:39:34 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:39:34 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 configure walinuxagent:amd64 2.1.3-0ubuntu4.1 <none>
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status unpacked walinuxagent:amd64 2.1.3-0ubuntu4.1
2016-09-21 22:40:46 status half-configured walinuxagent:amd64 2.1.3-0ubuntu4.1
ubuntu@jujuanj2kz35buz6c2lpu71qcuppgsd05hfhubbfqkspi144yl:~$ dpkg -l walinuxagent
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================================-========================-========================-====================================================================================
iF walinuxagent 2.1.3-0ubuntu4.1 amd64 Windows Azure Linux Agent

I have attached dmesg and cloud-init.log.

Revision history for this message
Paul Collins (pjdc) wrote :
Revision history for this message
Paul Collins (pjdc) wrote :
Revision history for this message
Scott Moser (smoser) wrote :

fwiw, this is actually a duplicate of bug 1576692.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.