Openstack's vendor_data2.json is not handled

Bug #1841104 reported by Marius L on 2019-08-22
32
This bug affects 4 people
Affects Status Importance Assigned to Milestone
cloud-init
Undecided
Unassigned

Bug Description

Starting with Newton, Openstack adds a vendor_data2.json in the metadata (based on the DynamicJSON vendor data provider).
However, cloud-init seems to not execute that.
According to the code (https://git.launchpad.net/cloud-init/tree/cloudinit/sources/helpers/openstack.py#n247), only the "vendor_data.json" is taken into account.

Paride Legovini (paride) wrote :

Thanks for your report. Apparently the NEWTON_TWO version was added explicitly to support vendor_data2.json:

https://git.launchpad.net/cloud-init/tree/cloudinit/sources/helpers/openstack.py#n50

but it seems the file is not actually being read.

Changed in cloud-init:
status: New → Triaged
Andrew Bogott (andrewbogott) wrote :

I'm still a bit confused by this... does this mean that dynamic vendor data cannot be used with cloud-init? If so it seems like a pretty big issue, rendering the whole DynamicJSON mostly useless.

James Falcon (falcojr) on 2021-02-05
Changed in cloud-init:
status: Triaged → Fix Committed

This bug is believed to be fixed in cloud-init in version 21.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Manish Mahalwal (mmahalwa) wrote :

Hi,
I am working with OpenStack Pike and cloud-init 21.1. I am able to successfully pass dynamic vendor data to the config drive of an instance. However, cloud-init 21.1 just reads all the 'x' bytes of the vendor_data2.json but it doesn't execute the contents of the json. Although, static vendor data works perfectly fine and the YAML file in the JSON is executed as expected by cloud-init 21.1

Relevant part from the cloud-init.log looks like this:

```
2021-04-09 10:51:47,104 - util.py[DEBUG]: Reading from /cfgdrive/openstack/2017-02-22/vendor_data2.json (quiet=False)
2021-04-09 10:51:47,114 - util.py[DEBUG]: Read 74 bytes from /cfgdrive/openstack/2017-02-22/vendor_data2.json
2021-04-09 10:52:02,332 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data.txt - wb: [600] 0 bytes
2021-04-09 10:52:02,335 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data.txt.i - wb: [600] 308 bytes
2021-04-09 10:52:02,336 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data2.txt - wb: [600] 0 bytes
2021-04-09 10:52:02,338 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data2.txt.i - wb: [600] 308 bytes
.
.
.
2021-04-09 10:52:02,352 - handlers.py[DEBUG]: start: init-network/consume-vendor-data2: reading and applying vendor-data2
2021-04-09 10:52:02,352 - stages.py[DEBUG]: no vendordata2 from datasource
```

It reads the 74 bytes but doesn't write the value (the YAML file) mapped to key "cloud-init" to the file to vendor-data2.txt, because it was not able to find "vendordata2 from datasource".

My vendor_data2.json looks like this:
`{"testing": {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}}`

Please let me know, if this issue is due to my JSON or an issue with how cloud-init handles vendor data. Since this pull request (https://github.com/canonical/cloud-init/pull/777) is fairly new, I am assuming not many people had the chance to test dynamic vendor data using cloud-init yet.

Andrew Bogott (andrewbogott) wrote :

Hello @mmahalwa!

Your json looks to have an extra layer around it. I believe that your issue is that you are feeding in your meta data with the testing@ name in your nova.conf.

As I interpret the metadata docs, testing@ would be metadata to be consumed by a service called 'testing' (or, realistically, not at all.) If I run a similar test with cloud-init@ instead it seems to work as expected.

From nova.conf:

[api]
vendordata_providers=DynamicJSON
vendordata_dynamic_targets=cloud-init@http://localhost:8780

On the affected VM:

curl http://169.254.169.254/openstack/latest/vendor_data2.json
{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - black\nfqdn: cloud-overridden-by-vendordata2.example.org."}

This seems like the right behavior to me but I'm open to suggestions if you think that the <name> value in nova.conf is intended to have a different meaning.

-Andrew

Manish Mahalwal (mmahalwa) wrote :

Thank you for your response @andrewboggot!

1. Can you confirm if the /var/lib/cloud/instances/<your-instance>/vendor-data2.txt contains the content of the YAML you passed?

2. When I set the name as cloud-init@<URL> in nova.conf, my custom REST service (to which Nova sends a POST request to) has to respond with a JSON string. The JSON string with which it responds is:
{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}

The openstack/latest/vendor_data2.json file becomes:
{"cloud-init": {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}}

This too is not parsed by cloud-init. If the REST service responds with only the YAML (the value pair) which is `#cloud-config\npackage_upgrade: True\npackages:\n - htop`, Nova refuses to accept this string because it is expecting a VALID JSON. So, I am unable to remove that extra layer in my JSON, to make it exactly like your JSON. I also tried leaving the `name@` field empty, but that doesn't work too.

>>So, how do I remove that extra layer of JSON to test if this works as expected?

On a different note, whether <name> value has a different meaning:
I may be wrong on this but, the intention of having the `name` field is just to have a name for the REST service from which the JSON came. For example, I have several REST services to query from which I have to fetch vendor data.

From (https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/vendordata-reboot.html)
"Each REST microservice is configured by appending a new item to the list provided in the vendordata_dynamic_targets flag. This flag is composed of entries of the form:"

vendordata_dynamic_targets=name1@http://example.com,name2@http://example2.com

The name element for these microservices becomes the new key for the vendor_data2.json file like,

{
    "name1": {
        "cloud-init": "#cloud-config\n..."
    },
    "name2": {
        "cloud-init": "#cloud-config\n..."
    }
}

Please note I have taken the above example from the Nova specs mentioned earlier and extended it for 2 REST services. Now, if we consider your approach of having cloud-init as the outer key, then how are we going to differentiate b/w several REST services.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers