Missing mandatory path: http://169.254.169.254/openstack/2018-08-27/meta_data.json

[Orestes Leal Rodriguez]

I'm running OpenStack Stein.

Cloud init from the bionic-cloud images it's not working correctly. I'm launching instances with ubuntu-bionic 18.04 cloud and the following message appears when reaching the openstack metadata server:

"[ 86.559374] cloud-init[843]: 2019-07-25 15:25:21,978 - util.py[WARNING]: Missing mandatory path: http://169.254.169.254/openstack/2018-08-27/meta_data.json"

Now, I can (without doing anything special) use ubuntu 14.04 cloud, 16.04 cloud, Centos7-cloud (all with different versions of cloud-init) and everything works, but with 'Cloud-init v. 19.1-1-gbaa47854-0ubuntu1~16.04.1' on the bionic cloud images it does not.

 I have tried to use EC2 datasources, OpenStack datasources [1] and it does not work. It connects to the metadata server and gets the meta_data.json file but doesn't do much more. therefore I'm unable to provision ssh keys, etc, with bionic-cloud. Attached is the full log for one bionic-cloud-1804 instance.

1. https://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html

[Dan Watkins (oddbloke)]

Hi Orestes,

Thanks for using cloud-init and for the bug report!

Are you able to capture the disk of a failing instance and pull files off of it? If so, could you attach cloud-init.log to this bug, please?

Could you also confirm that http://169.254.169.254/openstack/2018-08-27/meta_data.json does actually exist from within an instance?

Finally, 19.1-1-gbaa47854-0ubuntu1~16.04.1 is an unusual version to see on _bionic_ (I'd expect to see 18.04 there); was this a typo? If not, how are you constructing your images?

Thanks!

Dan

[Dan Watkins (oddbloke)]

(Once you've provided the above, please set the status back to New!)

[Orestes Leal Rodriguez (orestesleal)]

It was a copy/paste mistake, sorry.

Cloud-init v. 18.5-45-g3554ffe8-0ubuntu1~18.04.1

Attached is a combined log (with marks for discerning the different systems) with:

 - console from a xenial instance (http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img)
 - console from a bionic instance (http://cloud-images.ubuntu.com/bionic/20190720/bionic-server-cloudimg-amd64.img)

- both cloud-init.log for both vms

Note that note that one of them fetch keys, etc. from the metadata server on OpenStack but the other doesn't.

[Dan Watkins (oddbloke)]

Thanks for the extra info! It looks to me a lot like your OpenStack deployment is misconfigured. In both xenial and bionic, the metadata service appears to advertise support for metadata version 2018-08-27, but http://169.254.169.254/openstack/2018-08-27/meta_data.json is apparently not returning anything. I would suggest that you talk to your administrator to address this issue.

The reason that this isn't a hard blocker on xenial is that in xenial, we will try every data source that cloud-init ships. If you read through the log, you can see that we actually end up using the Ec2 data source, which hits the OpenStack EC2 compatibility API.

On bionic, cloud-init only attempts to reach out to network endpoints for platforms that it has positively identified it is running on. In this case, cloud-init has correctly identified that this is an OpenStack instance, so it only runs the OpenStack data source (which fails, due to the broken nature of your cloud's metadata API).

[Orestes Leal Rodriguez (orestesleal)]

Also note that when you see a public key being added on bionic that is because I put it on /etc/cloud/cloud.cfg to be able to login since it cannot use the metadata service. Also on bionic clou-init is "Cloud-init v.18.5-45-g3554ffe8-0ubuntu1~18.04.1" and on xenial is "Cloud-init v.19.1-1-gbaa47854-0ubuntu1~16.04.1". The one on xenial is more recent.

Orestes

[Dan Watkins (oddbloke)]

Hi Orestes,

To work around this problem, you'll need to configure ds-identify[0] to change its defaults. You should be able to do this by creating a file named /etc/cloud/ds-identify.cfg and putting this in there:

datasource: Ec2

This will configure ds-identify to disable its usual platform-checking behaviour and instead configure cloud-init to just attempt the Ec2 data source.

As this is an issue with the OpenStack deployment you're using (rather than a cloud-init bug), I'm going to close this out as Invalid. Please feel free to follow up here, or drop in to #cloud-init on freenode for any further assistance.

Thanks!

Dan

[0] https://git.launchpad.net/cloud-init/tree/tools/ds-identify

[Orestes Leal Rodriguez (orestesleal)]

Hi Guys,

I've fixed the issue, you all were correct, the problem was on the OpenStack side.
The problem was with the MTU (a device in the middle of those link don't allow >1500 MTU, we use jumbo frames.

But I have a question. Why xenial worked, did it transfer the metadata in chunks (therefore each part (json payload+protocol overhead) totals <1500 bytes)? because even with the MTU 9000 and the device discarding jumbo frames in the middle cloud-init on xenial worked (yes, with the EC2 datasource).

I would appreciate an insight of how that cloud init version works in this sense.

[Dan Watkins (oddbloke)]

Hi Orestes, cloud-init really just relies on libraries for its HTTP requests; we aren't doing anything out of the ordinary there. So presumably the responses from the EC2 compatibility layer are different in some way from the ones from /openstack/, but I don't believe that the way that cloud-init fetches each of them is all that different.

[Orestes Leal Rodriguez (orestesleal)]

Dan,
I appreciate your response, thanks a lot!

Orestes.

On 7/29/19, Dan Watkins <email address hidden> wrote:
> Hi Orestes, cloud-init really just relies on libraries for its HTTP
> requests; we aren't doing anything out of the ordinary there. So
> presumably the responses from the EC2 compatibility layer are different
> in some way from the ones from /openstack/, but I don't believe that the
> way that cloud-init fetches each of them is all that different.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1837927
>
> Title:
> Missing mandatory path:
> http://169.254.169.254/openstack/2018-08-27/meta_data.json
>
> Status in cloud-init:
> Invalid
>
> Bug description:
> I'm running OpenStack Stein.
>
> Cloud init from the bionic-cloud images it's not working correctly.
> I'm launching instances with ubuntu-bionic 18.04 cloud and the
> following message appears when reaching the openstack metadata server:
>
> "[ 86.559374] cloud-init[843]: 2019-07-25 15:25:21,978 -
> util.py[WARNING]: Missing mandatory path:
> http://169.254.169.254/openstack/2018-08-27/meta_data.json"
>
> Now, I can (without doing anything special) use ubuntu 14.04 cloud,
> 16.04 cloud, Centos7-cloud (all with different versions of cloud-init)
> and everything works, but with 'Cloud-init v.
> 19.1-1-gbaa47854-0ubuntu1~16.04.1' on the bionic cloud images it does
> not.
>
> I have tried to use EC2 datasources, OpenStack datasources [1] and it
> does not work. It connects to the metadata server and gets the
> meta_data.json file but doesn't do much more. therefore I'm unable to
> provision ssh keys, etc, with bionic-cloud. Attached is the full log
> for one bionic-cloud-1804 instance.
>
>
> 1.
> https://cloudinit.readthedocs.io/en/latest/topics/datasources/openstack.html
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1837927/+subscriptions
>

[Scott Moser (smoser)]

> But I have a question. Why xenial worked, did it transfer the metadata
> in chunks (therefore each part (json payload+protocol overhead) totals
> <1500 bytes)? because even with the MTU 9000 and the device discarding
> jumbo frames in the middle cloud-init on xenial worked (yes, with the
> EC2 datasource).
>
> I would appreciate an insight of how that cloud init version works in
> this sense.

I suspect the difference has to do with how networking is being
configured. That could either be a difference in cloud-init doing the
network config in "ephemeral mode" (it brings up a dhclient on the
first ethernet device) or a difference in ifupdown (and dhclient)
compared to netplan (and systemd-networkd).

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3412