A new feature in cloud-init identified possible datasources for this system as:['Ec2', 'None']. However, the datasource used was: Azure

[Mohammed Sameer B]

Whenever I login to server i get the below error message :
**************************************************************************
# A new feature in cloud-init identified possible datasources for #
# this system as: #
# ['Ec2', 'None'] #
# However, the datasource used was: Azure #
# #
# In the future, cloud-init will only attempt to use datasources that #
# are identified or specifically configured. #
# For more information see #
# https://bugs.launchpad.net/bugs/1669675 #
# #
# If you are seeing this message, please file a bug against #
# cloud-init at #
# https://bugs.launchpad.net/cloud-init/+filebug?field.tags=dsid #
# Make sure to include the cloud provider your instance is #
# running on. #
# #
# After you have filed a bug, you can disable this warning by launching #
# your instance with the cloud-config below, or putting that content #
# into /etc/cloud/cloud.cfg.d/99-warnings.cfg #
# #
# #cloud-config #
# warnings: #
# dsid_missing_source: off #
**************************************************************************

Disable the warnings above by:
  touch /root/.cloud-warnings.skip
or
  touch /var/lib/cloud/instance/warnings/.skip

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: cloud-init 0.7.9-48-g1c795b9-0ubuntu1~16.04.1
ProcVersionSignature: Ubuntu 4.4.0-174.204-generic 4.4.211
Uname: Linux 4.4.0-174-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.22
Architecture: amd64
Date: Thu Mar 19 09:58:34 2020
Ec2AMI: ami-08df7c3f25163f2b4
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-south-1a
Ec2InstanceType: m5.xlarge
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
PackageArchitecture: all
ProcEnviron:
 LC_CTYPE=UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install)

[Mohammed Sameer B (sameer-mohammed)]

The above mentioned server was freshly migrated from Azure CLoud to AWS cloud

[Scott Moser (smoser)]

Hi Mohammed,
Please attach the output of 'cloud-init collect-logs' and then set this back to New.

Thanks.

[Mohammed Sameer B (sameer-mohammed)]

Hi Scott,

As requested, I have added the output.

Thanks.

[Scott Moser (smoser)]

@Mohammed, I would do a 'rm -Rf /var/lib/waagent' before migrating future images. And I'd suggest that you can probably just 'touch /var/lib/cloud/instance/warnings/.skip'. But your system is in a wierd state.

@Cloud-init devs,

What happened here was:
 a.) instance booted on Azure
 b.) instance captured and moved to EC2
 c.) new instance on EC2 had:
   1. ds-identify recognize that it was on EC2
   2. python code path use the old Azure content (seed=/var/lib/waagent).
 d.) since c.1 and c.2 differed, cloud-init complained.

Added to that set of events was that this cloud-init iidentifies itself as 0.7.9. cloud-init version 17.1 (17.1-17-g45d361cb-0ubuntu1_16.04.1) was released to xenial in October of 2017.
So this system has not been updated in at least 2 years.

I'm not sure if this is bug is still present or not, and I believe that it would not be seen if ds-identify was enabled. But it does seem that in this old version there is discrepency in ds-identify detection of azure and the python path.

[Mohammed Sameer B (sameer-mohammed)]

Hi Scott,

I removed waagent, ran this command 'rm -Rf /var/lib/waagent' and upgraded cloud-init to 19.1.

But after reboot, I am unable to login to the server itself.

Can you suggest a way to fix cloudinit data source identification?

Thanks

[Scott Moser (smoser)]

@Mohammed,

If you are using an EBS root, then you can detach the volume and attach to another instance and then collect logs from it.

I might suggest that you make a snapshot, then change the volume to and add an ssh key to root's ssh keys and then re-attach that volume to the original instance and start it back up.

Its hard for me to guess what might be wrong though at this point.

Scott

[Mohammed Sameer B (sameer-mohammed)]

@Scott,

I have the image of server before making that change.

So Can you assist me on how to fix cloud-init data source identification?

Thanks

[Scott Moser (smoser)]

I'd suggest making the changes, and inserting a ssh key that you have
access to into root.
Then reboot and see what happens. You should then be able to log in
and debug a bit, run 'cloud-init collect-logs' and post here.

On Fri, Mar 20, 2020 at 10:20 AM Mohammed Sameer B
<email address hidden> wrote:
>
> @Scott,
>
> I have the image of server before making that change.
>
> So Can you assist me on how to fix cloud-init data source
> identification?
>
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to cloud-
> init in Ubuntu.
> https://bugs.launchpad.net/bugs/1868077
>
> Title:
> A new feature in cloud-init identified possible datasources for this
> system as:['Ec2', 'None']. However, the datasource used was: Azure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1868077/+subscriptions

[Mohammed Sameer B (sameer-mohammed)]

@Scott,

I have updated cloud-init to 19.4 version. But still I am facing the same issue.

Attached cloud-init collect-logs output for your reference.

Kindly help in fixing the same.

Thanks.

[Scott Moser (smoser)]

@Mohammed,
the cloud-init.log clearly shows cloud-init still using the Azure datasource due to the presense of /var/lib/waagent

I would suggest removing that directory.

[Mohammed Sameer B (sameer-mohammed)]

@Scott,

I have removed walinuxagent and deleted /var/lib/waagent. But when I restart the server, again I am facing the issue. I have attached cloud-init collect-logs output for our reference.

Thanks

[Scott Moser (smoser)]

It really seem slike you still have /var/lib/waagent around.
  See lines in your log like:
2020-03-21 07:36:37,988 - stages.py[DEBUG]: restored from checked
cache:ataSourceAzure [seed=/var/lib/waagent]

For reference, could youp lease attach:
   tar -C /var/lib/ -cvf var-lib-cloud.tar.gz cloud/

The instance should *not* have "restored from checked cache". The
above will help us to see what went wrong there.

And then you should do:
  rm -Rf /var/lib/waagent
  cloud-init clean --logs --seed

then try again.
And for

On Sat, Mar 21, 2020 at 3:50 AM Mohammed Sameer B
<email address hidden> wrote:
>
> @Scott,
>
> I have removed walinuxagent and deleted /var/lib/waagent. But when I
> restart the server, again I am facing the issue. I have attached cloud-
> init collect-logs output for our reference.
>
>
> Thanks
>
> ** Attachment added: "cloud-init.tar.gz"
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1868077/+attachment/5339604/+files/cloud-init.tar.gz
>
> --
> You received this bug notification because you are subscribed to cloud-
> init in Ubuntu.
> https://bugs.launchpad.net/bugs/1868077
>
> Title:
> A new feature in cloud-init identified possible datasources for this
> system as:['Ec2', 'None']. However, the datasource used was: Azure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1868077/+subscriptions

[Mohammed Sameer B (sameer-mohammed)]

@Scott,

i think it is now working fine after running this command "cloud-init clean --logs --seed".

I have attached cloud-init collect-logs output and "tar -C /var/lib/ -cvf var-lib-cloud.tar.gz cloud/". Can you please check and confirm that everything is working fine?

[Mohammed Sameer B (sameer-mohammed)]

++ PFA cloud-init collect-logs

[Scott Moser (smoser)]

@Mohammed,
I was hoping for /var/lib/cloud contents *before* running 'cloud-init clean'.

The other logs look fine.

@Other cloud-init devs,

I'd like someone else to hazard a guess at what went on here.
The general issue is:
 * very old cloud-init (0.7.9) booted on azure
 * migrated to ec2, still thought it was on azure
 * upgraded and it still happened
 * cloud-init clean --logs --seed and rm -Rf /var/lib/waagent fixed it.

I don't see in code how the issue shoudl have occurred after upgrade. but two things stick out:

 a.) _is_viable_platform will attempt to log an event even if not on azure. maybe that shouldnt happen.
 b.) _is_viable_platform will return true even if asset_tag failed, but /var/lib/waagent/ovf-env.xml existed.

that seems wrong

[Mohammed Sameer B (sameer-mohammed)]

@Scott,

So shall I consider the issue fixed or still some troubleshooting needs to be done?

Thanks

[Scott Moser (smoser)]

@Mohammed, Thank you for your help. I dont think we need anything
else from you.
Scott

On Mon, Mar 23, 2020 at 2:10 PM Mohammed Sameer B
<email address hidden> wrote:
>
> @Scott,
>
> So shall I consider the issue fixed or still some troubleshooting needs
> to be done?
>
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to cloud-
> init in Ubuntu.
> https://bugs.launchpad.net/bugs/1868077
>
> Title:
> A new feature in cloud-init identified possible datasources for this
> system as:['Ec2', 'None']. However, the datasource used was: Azure
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1868077/+subscriptions

[Dan Watkins (oddbloke)]

> a.) _is_viable_platform will attempt to log an event even if not on azure. maybe that shouldnt happen.

Maybe!

> b.) _is_viable_platform will return true even if asset_tag failed, but /var/lib/waagent/ovf-env.xml existed.

I don't _know_ that this is the reason, but is it possible that the Azure ARM instances don't have the chassis tag available?

[Dan Watkins (oddbloke)]

(Marking this Triaged/Low, as we're talking about cleaning things up, rather than fixing an issue that we know of, I believe.)