Runs on OpenStack, doesn't use OpenStack metadata, then complains

[Florian Haas]

Getting this on a Xenial image running 0.7.9-48-g1c795b9-0ubuntu:

**************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# ['OpenStack', 'None'] #
# However, the datasource used was: Ec2 #
# #
# In the future, cloud-init will only attempt to use datasources that #
# are identified or specifically configured. #
# For more information see #
# https://bugs.launchpad.net/bugs/1669675 #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# After you have filed a bug, you can disable this warning by launching #
# your instance with the cloud-config below, or putting that content #
# into /etc/cloud/cloud.cfg.d/99-warnings.cfg #
# #
# #cloud-config #
# warnings: #
# dsid_missing_source: off #
**************************************************************************
**************************************************************************
# This system is using the EC2 Metadata Service, but does not appear to #
# be running on Amazon EC2 or one of cloud-init's known platforms that #
# provide a EC2 Metadata service. In the future, cloud-init may stop #
# reading metadata from the EC2 Metadata Service unless the platform can #
# be identified. #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# For more information see #
# https://bugs.launchpad.net/bugs/1660385 #
# #
# After you have filed a bug, you can disable this warning by #
# launching your instance with the cloud-config below, or #
# putting that content into #
# /etc/cloud/cloud.cfg.d/99-ec2-datasource.cfg #
# #
# #cloud-config #
# datasource: #
# Ec2: #
# strict_id: false #
**************************************************************************

Disable the warnings above by:
  touch /home/ubuntu/.cloud-warnings.skip
or
  touch /var/lib/cloud/instance/warnings/.skip

However:

# /usr/lib/cloud-init/ds-identify --force
# cat /run/cloud-init/ds-identify.log

[up 1999.73s] ds-identify --force
policy loaded: mode=report report=false found=all maybe=all notfound=enabled
/etc/cloud/cloud.cfg.d/90_dpkg.cfg set datasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Ec2, CloudStack, None ]
DMI_PRODUCT_NAME=OpenStack Nova
DMI_SYS_VENDOR=OpenStack Foundation
DMI_PRODUCT_SERIAL=2c3b31d8-a9e3-446d-9964-554eb4ffc183
DMI_PRODUCT_UUID=321B9885-C76D-4BB5-988D-0CBA6662859B
PID_1_PLATFORM=unavailable
FS_LABELS=cloudimg-rootfs
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-4.4.0-70-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
VIRT=kvm
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=4.4.0-70-generic
UNAME_KERNEL_VERSION=#91-Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017
UNAME_MACHINE=x86_64
UNAME_NODENAME=mybox
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=NoCloud ConfigDrive OpenNebula DigitalOcean Azure AltCloud OVF MAAS GCE OpenStack CloudSigma SmartOS Ec2 CloudStack None
MODE=report
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=enabled
pid=1583 ppid=1560
is_container=false
check for 'OpenStack' returned found
ec2 platform is 'Unknown'.
check for 'Ec2' returned maybe
Found single datasource: OpenStack
[up 1999.74s] returning 0

So what gives? ds-identify detects that it's running on OpenStack, and identifies that as the only available datasource. Why does it use EC2 and spit out that warning?

[Scott Moser (smoser)]

Hi,
Florian,
Thanks for filing the bug.

I'm kind of confused also.

Some questions:
a.) please collect /var/log/cloud* in a tarball and attach
b.) Then, after doing so:
    # you should get a new cloud-init (0.7.9-90-g61eb03fe-0ubuntu1~16.04.1 in xenial)
    sudo apt-get update && sudo apt-get install cloud-init
    sudo mv /var/lib/cloud var-lib-cloud.old
    reboot

Does it still show up?

[Florian Haas (fghaas)]

Hi Scott, thanks for getting back to me. I can no longer reproduce this (with 0.7.9-48-g1c795b9), which baffles me even more. The only change that I am aware of that was made in the OpenStack configuration was that dnsmasq_local_resolv was enabled in Neutron's dhcp_agent.ini, and that two upstream DNS servers were added to the dnsmasq_dns_servers list as well.

I have absolutely no idea how that would have any bearing on cloud-init; after all, it just fetches from 169.254.169.254 and doesn't rely on name resolution at all (to the best of my knowledge). Would you have an idea why failure to resolve any DNS name(s) would cause cloud-init datasource detection breakage?

[Scott Moser (smoser)]

Florian,
Could you try switching it back, and then collecting /var/log/cloud-init.log ? journalctl output would also be appreciated.

If that reproduces the issue, then we can dig further. I don't honestly think it should, but i'd be interested in seeing.

I'll just stick this in 'incomplete', if you reproduce again, please collect that info and put it into 'Confirmed'.

Scott

[Joshua Schwartz (jschwartz)]

I am also experiencing this issue. The problem is that it's not alway 100% reproducible. I see it when deploying some instances and not on others. I can shorten the datasources list to only include OpenStack which will ensure it always works, but I would rather that it only uses the detected datasources and errors out otherwise.

The current behavior is that if it initially comes up using Ec2 it generates a set of ssh host keys, later the system is rebooted and it uses the OpenStack datasource which generates a new set of ssh host keys.

I'm running version 0.7.9-233 on Xenial.

I have attached the requested logs.

[James Falcon (falcojr)]

Given that this was filed against such an old version of cloud-init, and there are no currently known instances of this happening, I'm going to close this as won't fix. If you're still seeing this issue with the current version of cloud-init, please file a new bug with updated logs.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/2873