stuck "running" on DataSourceNone

[Sirio Balmelli]

On a bare-metal machine with `None` datasource:

```
datasource_list:
  - None
datasource:
  None:
```

The 'config' module does not terminate:

```
$ sudo cloud-init modules
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'modules:config' at Thu, 03 Sep 2020 14:01:22 +0000. Up 5526.72 seconds.
```

My cloud-init file seems valid:

```
$ cloud-init devel schema --config-file /etc/cloud/cloud.cfg
Valid cloud-config file /etc/cloud/cloud.cfg
```

Setting `cloud_config_modules` to an empty list `cloud_config_modules: []` gives:

```
$ sudo cloud-init modules
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'modules:config' at Thu, 03 Sep 2020 15:16:31 +0000. Up 150.15 seconds.
No 'config' modules to run under section 'cloud_config_modules'
```

Possibly related to https://bugs.launchpad.net/cloud-init/+bug/1694399 but uncertain.

This is my `status.json`:

```
{
 "v1": {
  "datasource": "DataSourceNone",
  "init": {
   "errors": [],
   "finished": 1599134018.9847271,
   "start": 1599134018.7709196
  },
  "init-local": {
   "errors": [],
   "finished": 1599133970.417612,
   "start": 1599133970.1241682
  },
  "modules-config": {
   "errors": [],
   "finished": 1599134019.5227249,
   "start": 1599134019.4342222
  },
  "modules-final": {
   "errors": [],
   "finished": null,
   "start": null
  },
  "modules-init": {
   "errors": [],
   "finished": null,
   "start": null
  },
  "stage": null
 }
}
```

My apport is:

```
ProblemType: Bug
ApportVersion: 2.20.1-0ubuntu2.24
Architecture: amd64
Date: Thu Sep 3 14:09:34 2020
Dependencies:
 adduser 3.113+nmu3ubuntu4
 apt 1.2.32ubuntu0.1
 apt-utils 1.2.32ubuntu0.1
 ca-certificates 20190110~16.04.1
 cloud-guest-utils 0.27-0ubuntu25.1
 coreutils 8.25-2ubuntu3~16.04
 cron 3.0pl1-128ubuntu2
 dbus 1.10.6-1ubuntu3.6
 debconf 1.5.58ubuntu2
 debconf-i18n 1.5.58ubuntu2
 debianutils 4.7
 dh-python 2.20151103ubuntu1.2
 distro-info-data 0.28ubuntu0.14
 dpkg 1.18.4ubuntu1.6
 e2fslibs 1.42.13-1ubuntu1.2
 e2fsprogs 1.42.13-1ubuntu1.2
 eatmydata 105-3
 file 1:5.25-2ubuntu1.4
 gcc-5-base 5.4.0-6ubuntu1~16.04.12
 gcc-6-base 6.0.1-0ubuntu1
 gdisk 1.0.1-1build1
 gir1.2-glib-2.0 1.46.0-3ubuntu1
 gnupg 1.4.20-1ubuntu3.3
 gpgv 1.4.20-1ubuntu3.3
 groff-base 1.22.3-7
 ifupdown 0.8.10ubuntu1.4
 init-system-helpers 1.29ubuntu4
 initscripts 2.88dsf-59.3ubuntu2
 insserv 1.14.0-5ubuntu3
 iproute2 4.3.0-1ubuntu3.16.04.5
 isc-dhcp-client 4.3.3-5ubuntu12.10
 isc-dhcp-common 4.3.3-5ubuntu12.10
 iso-codes 3.65-1
 krb5-locales 1.13.2+dfsg-5ubuntu2.1
 libacl1 2.2.52-3
 libapparmor1 2.10.95-0ubuntu2.11
 libapt-inst2.0 1.2.32ubuntu0.1
 libapt-pkg5.0 1.2.32ubuntu0.1
 libasn1-8-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libattr1 1:2.4.47-2
 libaudit-common 1:2.4.5-1ubuntu2.1
 libaudit1 1:2.4.5-1ubuntu2.1
 libblkid1 2.27.1-6ubuntu3.10
 libbz2-1.0 1.0.6-8ubuntu0.2
 libc6 2.23-0ubuntu11.2
 libcap-ng0 0.7.7-1
 libcomerr2 1.42.13-1ubuntu1.2
 libcurl3-gnutls 7.47.0-1ubuntu2.16
 libdb5.3 5.3.28-11ubuntu0.2
 libdbus-1-3 1.10.6-1ubuntu3.6
 libdbus-glib-1-2 0.106-1
 libdns-export162 1:9.10.3.dfsg.P4-8ubuntu1.17
 libeatmydata1 105-3
 libexpat1 2.1.0-7ubuntu0.16.04.5
 libfdisk1 2.27.1-6ubuntu3.10
 libffi6 3.2.1-4
 libgcc1 1:6.0.1-0ubuntu1
 libgcrypt20 1.6.5-2ubuntu0.6
 libgirepository-1.0-1 1.46.0-3ubuntu1
 libglib2.0-0 2.48.2-0ubuntu4.6
 libglib2.0-data 2.48.2-0ubuntu4.6
 libgmp10 2:6.1.0+dfsg-2
 libgnutls30 3.4.10-4ubuntu1.8
 libgpg-error0 1.21-2ubuntu1
 libgpm2 1.20.4-6.1
 libgssapi-krb5-2 1.13.2+dfsg-5ubuntu2.1
 libgssapi3-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libhcrypto4-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libheimbase1-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libheimntlm0-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libhogweed4 3.2-1ubuntu0.16.04.1
 libhx509-5-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libicu55 55.1-7ubuntu0.5
 libidn11 1.32-3ubuntu1.2
 libisc-export160 1:9.10.3.dfsg.P4-8ubuntu1.17
 libk5crypto3 1.13.2+dfsg-5ubuntu2.1
 libkeyutils1 1.5.9-8ubuntu1
 libkrb5-26-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libkrb5-3 1.13.2+dfsg-5ubuntu2.1
 libkrb5support0 1.13.2+dfsg-5ubuntu2.1
 libldap-2.4-2 2.4.42+dfsg-2ubuntu3.9
 liblocale-gettext-perl 1.07-1build1
 liblz4-1 0.0~r131-2ubuntu2
 liblzma5 5.1.1alpha+20120614-2ubuntu2
 libmagic1 1:5.25-2ubuntu1.4
 libmnl0 1.0.3-5
 libmount1 2.27.1-6ubuntu3.10
 libmpdec2 2.4.2-1
 libncurses5 6.0+20160213-1ubuntu1
 libncursesw5 6.0+20160213-1ubuntu1
 libnettle6 3.2-1ubuntu0.16.04.1
 libp11-kit0 0.23.2-5~ubuntu16.04.1
 libpam-modules 1.1.8-3.2ubuntu2.1
 libpam-modules-bin 1.1.8-3.2ubuntu2.1
 libpam-runtime 1.1.8-3.2ubuntu2.1
 libpam0g 1.1.8-3.2ubuntu2.1
 libpcre3 2:8.38-3.1
 libpopt0 1.16-10
 libprocps4 2:3.3.10-4ubuntu2.5
 libpython-stdlib 2.7.12-1~16.04
 libpython2.7-minimal 2.7.12-1ubuntu0~16.04.12
 libpython2.7-stdlib 2.7.12-1ubuntu0~16.04.12
 libpython3-stdlib 3.5.1-3
 libpython3.5-minimal 3.5.2-2ubuntu0~16.04.11
 libpython3.5-stdlib 3.5.2-2ubuntu0~16.04.11
 libreadline6 6.3-8ubuntu2
 libroken18-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 librtmp1 2.4+20151223.gitfa8646d-1ubuntu0.1
 libsasl2-2 2.1.26.dfsg1-14ubuntu0.2
 libsasl2-modules 2.1.26.dfsg1-14ubuntu0.2
 libsasl2-modules-db 2.1.26.dfsg1-14ubuntu0.2
 libselinux1 2.4-3build2
 libsemanage-common 2.3-1build3
 libsemanage1 2.3-1build3
 libsepol1 2.4-2
 libsmartcols1 2.27.1-6ubuntu3.10
 libsqlite3-0 3.11.0-1ubuntu1.5
 libss2 1.42.13-1ubuntu1.2
 libssl1.0.0 1.0.2g-1ubuntu4.16
 libstdc++6 5.4.0-6ubuntu1~16.04.12
 libsystemd0 229-4ubuntu21.28
 libtasn1-6 4.7-3ubuntu0.16.04.3
 libtext-charwidth-perl 0.04-7build5
 libtext-iconv-perl 1.7-5build4
 libtext-wrapi18n-perl 0.06-7.1
 libtinfo5 6.0+20160213-1ubuntu1
 libudev1 229-4ubuntu21.28
 libusb-0.1-4 2:0.1.12-28
 libustr-1.0-1 1.0.4-5
 libuuid1 2.27.1-6ubuntu3.10
 libwind0-heimdal 1.7~git20150920+dfsg-4ubuntu1.16.04.1
 libxml2 2.9.3+dfsg1-1ubuntu0.7
 libxtables11 1.6.0-2ubuntu3
 libyaml-0-2 0.1.6-3
 lsb-base 9.20160110ubuntu0.2
 lsb-release 9.20160110ubuntu0.2
 mime-support 3.59ubuntu1
 mount 2.27.1-6ubuntu3.10
 multiarch-support 2.23-0ubuntu11.2
 openssl 1.0.2g-1ubuntu4.16
 passwd 1:4.2-3.1ubuntu5.4
 perl-base 5.22.1-9ubuntu0.6
 powermgmt-base 1.31+nmu1
 procps 2:3.3.10-4ubuntu2.5
 psmisc 22.21-2.1ubuntu0.1
 python 2.7.12-1~16.04
 python-apt-common 1.1.0~beta1ubuntu0.16.04.9
 python-minimal 2.7.12-1~16.04
 python2.7 2.7.12-1ubuntu0~16.04.12
 python2.7-minimal 2.7.12-1ubuntu0~16.04.12
 python3 3.5.1-3
 python3-apt 1.1.0~beta1ubuntu0.16.04.9
 python3-blinker 1.3.dfsg2-1build1
 python3-chardet 2.3.0-2
 python3-configobj 5.0.6-2
 python3-cryptography 1.2.3-1ubuntu0.2
 python3-dbus 1.2.0-3
 python3-gi 3.20.0-0ubuntu1
 python3-idna 2.0-3
 python3-jinja2 2.8-1ubuntu0.1
 python3-json-pointer 1.9-3
 python3-jsonpatch 1.19-3
 python3-jwt 1.3.0-1ubuntu0.1
 python3-markupsafe 0.23-2build2
 python3-minimal 3.5.1-3
 python3-oauthlib 1.0.3-1
 python3-pkg-resources 20.7.0-1
 python3-pyasn1 0.1.9-1
 python3-pycurl 7.43.0-1ubuntu1
 python3-requests 2.9.1-3ubuntu0.1
 python3-serial 3.0.1-1
 python3-six 1.10.0-3
 python3-software-properties 0.96.20.10
 python3-urllib3 1.13.1-2ubuntu0.16.04.3
 python3-yaml 3.11-3build1
 python3.5 3.5.2-2ubuntu0~16.04.11
 python3.5-minimal 3.5.2-2ubuntu0~16.04.11
 readline-common 6.3-8ubuntu2
 sed 4.2.2-7
 sensible-utils 0.0.9ubuntu0.16.04.1
 sgml-base 1.26+nmu4ubuntu1
 shared-mime-info 1.5-2ubuntu0.2
 software-properties-common 0.96.20.10
 sysv-rc 2.88dsf-59.3ubuntu2
 sysvinit-utils 2.88dsf-59.3ubuntu2
 tar 1.28-2.1ubuntu0.1
 ubuntu-keyring 2012.05.19.1
 ucf 3.0036
 unattended-upgrades 1.1ubuntu1.18.04.7~16.04.6
 update-motd 3.6-0ubuntu1
 util-linux 2.27.1-6ubuntu3.10
 uuid-runtime 2.27.1-6ubuntu3.10
 xdg-user-dirs 0.15-2ubuntu6.16.04.1
 xml-core 0.13+nmu2
 xz-utils 5.1.1alpha+20120614-2ubuntu2
 zlib1g 1:1.2.8.dfsg-2ubuntu4.3
DistroRelease: Ubuntu 16.04
InstallationDate: Installed on 2016-04-11 (1606 days ago)
InstallationMedia: Ubuntu-Server 14.04.2 LTS "Trusty Tahr" - Release amd64 (20150218.1)
Package: cloud-init 20.2-45-g5f7825e2-0ubuntu1~16.04.1
PackageArchitecture: all
ProcCpuinfoMinimal:
 processor : 11
 vendor_id : GenuineIntel
 cpu family : 6
 model : 63
 model name : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
 stepping : 2
 microcode : 0x43
 cpu MHz : 1933.980
 cache size : 15360 KB
 physical id : 0
 siblings : 12
 core id : 5
 cpu cores : 6
 apicid : 11
 initial apicid : 11
 fpu : yes
 fpu_exception : yes
 cpuid level : 15
 wp : yes
 flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
 bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
 bogomips : 6598.53
 clflush size : 64
 cache_alignment : 64
 address sizes : 46 bits physical, 48 bits virtual
 power management:
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.4.0-189.219-generic 4.4.232
SourcePackage: cloud-init
Tags: xenial uec-images
Uname: Linux 4.4.0-189-generic x86_64
UpgradeStatus: Upgraded to xenial on 2018-09-12 (721 days ago)
_MarkForUpload: True
logs.tgz: Error: [Errno 2] No such file or directory: '/tmp/cloud-init-logs.tgz'
modified.conffile..etc.cloud.cloud.cfg: [modified]
mtime.conffile..etc.cloud.cloud.cfg: 2020-09-03T11:52:01.102495
```

Unfortunately I cannot attach cloud-init.tar because it contains sensitive infos (not my machines).

Any tips on how to further explore and/or fix this bug are much appreciated.

[Paride Legovini (paride)]

Hi Silvio and thanks for this bug report. When you run `cloud-init modules` you actually trigger the module execution, that's why you see that 'running'. The "Up 5526.72 seconds" part is just the uptime of your machine.

Running `cloud-init status` or `cloud-init status --long` should show you if cloud-init is done or not.

I'm setting the status of this report to Incomplete for the moment. If you agree with my analysis please set it to Invalid, otherwise comment back, set it to New, and we'll look at it again.

[Paride Legovini (paride)]

s/Silvio/Sirio/, sorry :-)

[Sirio Balmelli (siriobalmelli)]

Thank you very much for the fast response and for pointing out where I went wrong with `cloud-init modules`

The bug is very much still there however ... this same machine has been sitting for 3 days with no usage, and still shows "running" status:

```
$ cloud-init status -l
status: running
time: Thu, 03 Sep 2020 15:36:45 +0000
detail:
DataSourceNone
$ uptime
 14:39:01 up 3 days, 23:03, 1 user, load average: 0.10, 0.04, 0.01
```

Another interesting datum is that `analyze blame` actually crashes (might not be related):

```
$ sudo cloud-init analyze blame
Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 9, in <module>
    load_entry_point('cloud-init==20.2', 'console_scripts', 'cloud-init')()
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main
    get_uptime=True, func=functor, args=(name, args))
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2620, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/__main__.py", line 158, in analyze_blame
    blame_format)):
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/show.py", line 374, in show_events
    return generate_records(events, print_format=print_format)
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/show.py", line 348, in generate_records
    prev_evt = unprocessed.pop()
IndexError: pop from empty list
```

[Dan Watkins (oddbloke)]

Hi Sirio,

Thanks for using cloud-init, and for filing this bug!

cloud-init expects that its four boot phases will (a) all run, and (b) run in a particular order; see https://cloudinit.readthedocs.io/en/latest/topics/boot.html for further details.

Specifically, here, you haven't run `modules-final`, which means that `/run/cloud-init/result.json` is not written (https://github.com/canonical/cloud-init/blob/master/cloudinit/cmd/main.py#L673-L682), which means that `cloud-init status` will always return "running" (https://github.com/canonical/cloud-init/blob/master/cloudinit/cmd/status.py#L120-L122).

I don't know why `analyze blame` is crashing. It looks to me like the first event in the log being processed is not a start event; this is a precondition for the code to work at all, so I'm guessing it's due to partial running of cloud-init (or perhaps multiple runs when debugging). Regardless, we can improve this error message. If you can paste the first few lines of /var/log/cloud-init.log (provided they don't include private info, of course!), we might be able to make some progress on it. I think it probably makes sense to handle that as a separate bug, as it is at least partially orthogonal, would you mind filing it separately and including that information over there?

Finally: if you are just looking to execute a subset of cloud-init modules without hooking cloud-init into boot fully, you may want to look at using `cloud-init single` (https://cloudinit.readthedocs.io/en/latest/topics/cli.html#single) instead.

Thanks!

Dan

[Sirio Balmelli (siriobalmelli)]

Thank you for looking into this, and I appreciate you taking the time to give me more info on cloud-init.

As I currently understand it, `modules-final` are declared, but not being executed.

The relevant part of `cloud.cfg`:

```
final_message: my-hostname
cloud_init_modules:
  - bootcmd
  - ca-certs
  - rsyslog
  - seed_random
  - users-groups
  - set_hostname
  - update_etc_hosts
  - update_hostname
  - ssh
  - [write-files, always]
cloud_config_modules:
  - timezone
  - [runcmd, always]
cloud_final_modules:
  - [scripts-user, always]
  - [final-message, always]
system_info:
  distro: ubuntu
  paths:
    cloud_dir: /var/lib/cloud/
    templates_dir: /etc/cloud/templates/
    upstart_dir: /etc/init/
  ssh_svcname: ssh

datasource_list:
  - None
datasource:
  None:
```

Interestingly, I can execute each of the `config` and `final` modules using cloud-init single:

```
$ sudo cloud-init single --name timezone --frequency always
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'single' at Tue, 08 Sep 2020 10:39:13 +0000. Up 414200.69 seconds.

$ sudo cloud-init single --name runcmd --frequency always
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'single' at Tue, 08 Sep 2020 10:39:39 +0000. Up 414226.75 seconds.

$ sudo cloud-init single --name scripts-user --frequency always
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'single' at Tue, 08 Sep 2020 10:40:18 +0000. Up 414264.83 seconds.

# ... output elided, but returning 0

$ sudo cloud-init single --name final-message --frequency always
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'single' at Tue, 08 Sep 2020 10:40:49 +0000. Up 414296.47 seconds.
my-hostname
2020-09-08 10:40:49,790 - cc_final_message.py[WARNING]: Used fallback datasource
```

... and yet the system is still suck on "running" status:

```
$ cloud-init status -l
status: running
time: Thu, 03 Sep 2020 15:36:45 +0000
detail:
DataSourceNone
```

I'm curious about the message `[WARNING]: Used fallback datasource` above ... there is only one datasource and it's local, not sure why it would be using a fallback?

In case it helps, rebooting yields the same result:

```
$ cloud-init status -l
status: running
time: Tue, 08 Sep 2020 10:52:33 +0000
detail:
DataSourceNone
$ uptime
 11:41:45 up 49 min, 1 user, load average: 0.12, 0.04, 0.01
$ sudo cloud-init analyze blame
Traceback (most recent call last):
  File "/usr/bin/cloud-init", line 9, in <module>
    load_entry_point('cloud-init==20.2', 'console_scripts', 'cloud-init')()
  File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 893, in main
    get_uptime=True, func=functor, args=(name, args))
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2620, in log_time
    ret = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/__main__.py", line 158, in analyze_blame
    blame_format)):
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/show.py", line 374, in show_events
    return generate_records(events, print_format=print_format)
  File "/usr/lib/python3/dist-packages/cloudinit/analyze/show.py", line 348, in generate_records
    prev_evt = unprocessed.pop()
IndexError: pop from empty l...

[Dan Watkins (oddbloke)]

Hi Sirio,

Did you read the boot stages documentation? It's not clear from it, but each of those stages executes cloud-init once with different parameters: `cloud-init init --local`, `cloud-init init`, `cloud-init modules --mode=config`, `cloud-init modules --mode=final`.

If you are not executing all 4 of these in order (and, to some extent, at the expected point in boot), then you are going to see unexpected results. To address your specific questions, if you don't run the `init` stages then datasource discovery does not happen, and if you don't run the `modules-final` stage then `cloud_final_modules` are not executed (and so cloud-init will always display `running`).

Cheers,

Dan

[Sirio Balmelli (siriobalmelli)]

Hi Dan,

Thank you very much, help and additional info appreciated.

I think I'm tracking with the various stages - I can manually `clean` the machine and then run it through all stages and it says `done` ... but when I reboot it's stuck on `running` again:

```
$ sudo cloud-init clean
upop@:~$ sudo cloud-init init --local
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'init-local' at Wed, 09 Sep 2020 11:23:17 +0000. Up 88290.82 seconds.

$ sudo cloud-init init
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'init' at Wed, 09 Sep 2020 11:23:28 +0000. Up 88301.55 seconds.

[ output elided ]

$ sudo cloud-init modules --mode=config
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'modules:config' at Wed, 09 Sep 2020 11:23:43 +0000. Up 88316.73 seconds.

$ sudo cloud-init modules --mode=final
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'modules:final' at Wed, 09 Sep 2020 11:23:54 +0000. Up 88327.25 seconds.

[ script output elided: returns 0]

2020-09-09 11:24:17,205 - cc_final_message.py[WARNING]: Used fallback datasource
$ cloud-init status
status: done
$ sudo reboot

##
# ... wait for machine to come back up ...
##

$ cloud-init status -l
status: running
time: Wed, 09 Sep 2020 11:25:44 +0000
detail:
DataSourceNone
```

You mention executing the stages in order ... I believe this is being handled by systemd, using the service files installed by cloud-init, correct? This is what I see systemd doing:

```
$ sudo systemctl status cloud*
● cloud-init.service - Initial cloud-init job (metadata service crawler)
   Loaded: loaded (/lib/systemd/system/cloud-init.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2020-09-09 11:25:43 GMT; 7min ago
  Process: 2220 ExecStart=/usr/bin/cloud-init init (code=exited, status=0/SUCCESS)
 Main PID: 2220 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/cloud-init.service

Sep 09 11:25:43 systemd[1]: Started Initial cloud-init job (metadata service crawler).

● cloud-init-local.service - Initial cloud-init job (pre-networking)
   Loaded: loaded (/lib/systemd/system/cloud-init-local.service; enabled; vendor preset: enabled)
   Active: active (exited) since Wed 2020-09-09 11:24:55 GMT; 7min ago
  Process: 1068 ExecStart=/bin/touch /run/cloud-init/network-config-ready (code=exited, status=0/SUCCESS)
  Process: 481 ExecStart=/usr/bin/cloud-init init --local (code=exited, status=0/SUCCESS)
 Main PID: 1068 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/cloud-init-local.service

Sep 09 11:24:55 cloud-init[481]: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'init-local' at Wed, 09 Sep 2020 11:24:54 +0000. Up 3.09 seconds.
Sep 09 11:24:55 systemd[1]: Started Initial cloud-init job (pre-networking).
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

● cloud-init.target - Cloud-init target
   Loaded: loaded (/lib/systemd/system/cloud-init.target; static; vendor preset: enabled)
   Active: inactive (dead)

● cloud-final.service - Execute cloud user/final scripts
   Loaded: loaded (/lib/systemd/system/cloud-final.service; enabled; vendor preset: enabled)
   Active: inactive (dead)...

[Dan Watkins (oddbloke)]

Hi Sirio,

If you have the systemd units installed, why are you running cloud-init manually at all?

Can you include the contents of /run/cloud/ds-identify.log from the affected instance, please?

Thanks!

Dan

[Sirio Balmelli (siriobalmelli)]

Hi Dan,

Thanks for working me through this, I'm learning quite a bit here.

Re: running manually - this is to see the actual output of the stages when debugging this issue, otherwise cloud-init is executed by systemd on boot with no intervention.

For the record, the current workflow when setting up one of these machines is:

- write a template to /etc/cloud/cloud.cfg
- call `cloud-init clean && cloud init`
- reboot the machine

Here is `ds-identify.log`, please note the slightly different location than expected:

```
$ cat /run/cloud-init/ds-identify.log
[up 2.73s] ds-identify
policy loaded: mode=report report=false found=all maybe=all notfound=enabled
/etc/cloud/cloud.cfg set datasource_list:
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
DMI_PRODUCT_NAME=To Be Filled By O.E.M.
DMI_SYS_VENDOR=To Be Filled By O.E.M.
DMI_PRODUCT_SERIAL=To Be Filled By O.E.M.
DMI_PRODUCT_UUID=03000200-0400-0500-0006-000700080009
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=To Be Filled By O.E.M.
FS_LABELS=hostname:0,hostname:0
ISO9660_DEVS=
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.0-189-generic root=UUID=73550318-2c71-40ca-ab97-b841dd1b1d51 ro vga=normal nomodeset
VIRT=none
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.0-189-generic
UNAME_KERNEL_VERSION=#219-Ubuntu SMP Tue Aug 11 12:26:50 UTC 2020
UNAME_MACHINE=x86_64
UNAME_NODENAME=u2-hrhrtv-ngs
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
MODE=report
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=enabled
pid=446 ppid=432
is_container=false
is_ds_enabled(IBMCloud) = true.
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
is_ds_enabled(IBMCloud) = true.
1 datasources returned maybe: Ec2
[up 2.80s] returning 0
```

I'm curious that this outputs:

```
no datasource_list found, using default: MAAS ConfigDrive NoCloud AltCloud Azure Bigstep CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale RbxCloud
```

... but `/etc/cloud/cloud.cfg` lists:

```
datasource_list:
  - NoCloud
  - None
datasource:
  None:
```

This data source issue may or may not be related to cloud-init stuck at `running` ... but I did not find additional information outside of https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html

Thank you,

Sirio

[Sirio Balmelli (siriobalmelli)]

amendment re: data source:

There is no change in `/run/cloud-init/ds-identify.log` whether `/etc/cloud/cloud.cfg` contains:

```
datasource_list:
  - NoCloud
  - None
datasource:
  None:
```

or

```
datasource_list:
  - None
datasource:
  None:
```

[Sirio Balmelli (siriobalmelli)]

update: machine has been up since last reboot and cloud-init is still on `running`:

```
$ uptime
 07:39:01 up 10 days, 20:28, 1 user, load average: 0.00, 0.00, 0.00

$ cloud-init status
status: running

$ tail -f /var/log/cloud-init*
==> /var/log/cloud-init.log <==
2020-09-11 11:10:56,424 - schema.py[DEBUG]: Ignoring schema validation. python-jsonschema is not present
2020-09-11 11:10:56,424 - util.py[DEBUG]: Shellified 8 commands.
2020-09-11 11:10:56,424 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/iid-datasource-none/scripts/runcmd - wb: [700] 362 bytes
2020-09-11 11:10:56,425 - handlers.py[DEBUG]: finish: modules-config/config-runcmd: SUCCESS: config-runcmd ran successfully
2020-09-11 11:10:56,425 - main.py[DEBUG]: Ran 2 modules with 0 failures
2020-09-11 11:10:56,425 - atomic_helper.py[DEBUG]: Atomically writing to file /var/lib/cloud/data/status.json (via temporary file /var/lib/cloud/data/tmp06fgd5b8) - w: [644] 541 bytes/chars
2020-09-11 11:10:56,425 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2020-09-11 11:10:56,426 - util.py[DEBUG]: Read 13 bytes from /proc/uptime
2020-09-11 11:10:56,426 - util.py[DEBUG]: cloud-init mode 'modules' took 0.092 seconds (0.09)
2020-09-11 11:10:56,426 - handlers.py[DEBUG]: finish: modules-config: SUCCESS: running modules for config

==> /var/log/cloud-init-output.log <==
Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~16.04.1 running 'modules:config' at Fri, 11 Sep 2020 11:10:56 +0000. Up 52.64 seconds.
```

Any input or some further things to check would be much appreciated.

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3778