Openstack's vendor_data2.json is not handled

Bug #1841104 reported by Marius L
36
This bug affects 5 people
Affects Status Importance Assigned to Milestone
cloud-init
Fix Released
Undecided
Unassigned

Bug Description

Starting with Newton, Openstack adds a vendor_data2.json in the metadata (based on the DynamicJSON vendor data provider).
However, cloud-init seems to not execute that.
According to the code (https://git.launchpad.net/cloud-init/tree/cloudinit/sources/helpers/openstack.py#n247), only the "vendor_data.json" is taken into account.

Revision history for this message
Paride Legovini (paride) wrote :

Thanks for your report. Apparently the NEWTON_TWO version was added explicitly to support vendor_data2.json:

https://git.launchpad.net/cloud-init/tree/cloudinit/sources/helpers/openstack.py#n50

but it seems the file is not actually being read.

Changed in cloud-init:
status: New → Triaged
Revision history for this message
Andrew Bogott (andrewbogott) wrote :

I'm still a bit confused by this... does this mean that dynamic vendor data cannot be used with cloud-init? If so it seems like a pretty big issue, rendering the whole DynamicJSON mostly useless.

Revision history for this message
Richard Harding (rharding) wrote :
James Falcon (falcojr)
Changed in cloud-init:
status: Triaged → Fix Committed
Revision history for this message
Dan Watkins (oddbloke) wrote : Fixed in cloud-init version 21.1.

This bug is believed to be fixed in cloud-init in version 21.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
Revision history for this message
Manish Mahalwal (mmahalwa) wrote :

Hi,
I am working with OpenStack Pike and cloud-init 21.1. I am able to successfully pass dynamic vendor data to the config drive of an instance. However, cloud-init 21.1 just reads all the 'x' bytes of the vendor_data2.json but it doesn't execute the contents of the json. Although, static vendor data works perfectly fine and the YAML file in the JSON is executed as expected by cloud-init 21.1

Relevant part from the cloud-init.log looks like this:

```
2021-04-09 10:51:47,104 - util.py[DEBUG]: Reading from /cfgdrive/openstack/2017-02-22/vendor_data2.json (quiet=False)
2021-04-09 10:51:47,114 - util.py[DEBUG]: Read 74 bytes from /cfgdrive/openstack/2017-02-22/vendor_data2.json
2021-04-09 10:52:02,332 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data.txt - wb: [600] 0 bytes
2021-04-09 10:52:02,335 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data.txt.i - wb: [600] 308 bytes
2021-04-09 10:52:02,336 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data2.txt - wb: [600] 0 bytes
2021-04-09 10:52:02,338 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/74eeba86-84b8-4599-856e-88413029ebc8/vendor-data2.txt.i - wb: [600] 308 bytes
.
.
.
2021-04-09 10:52:02,352 - handlers.py[DEBUG]: start: init-network/consume-vendor-data2: reading and applying vendor-data2
2021-04-09 10:52:02,352 - stages.py[DEBUG]: no vendordata2 from datasource
```

It reads the 74 bytes but doesn't write the value (the YAML file) mapped to key "cloud-init" to the file to vendor-data2.txt, because it was not able to find "vendordata2 from datasource".

My vendor_data2.json looks like this:
`{"testing": {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}}`

Please let me know, if this issue is due to my JSON or an issue with how cloud-init handles vendor data. Since this pull request (https://github.com/canonical/cloud-init/pull/777) is fairly new, I am assuming not many people had the chance to test dynamic vendor data using cloud-init yet.

Revision history for this message
Andrew Bogott (andrewbogott) wrote :

Hello @mmahalwa!

Your json looks to have an extra layer around it. I believe that your issue is that you are feeding in your meta data with the testing@ name in your nova.conf.

As I interpret the metadata docs, testing@ would be metadata to be consumed by a service called 'testing' (or, realistically, not at all.) If I run a similar test with cloud-init@ instead it seems to work as expected.

From nova.conf:

[api]
vendordata_providers=DynamicJSON
vendordata_dynamic_targets=cloud-init@http://localhost:8780

On the affected VM:

curl http://169.254.169.254/openstack/latest/vendor_data2.json
{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - black\nfqdn: cloud-overridden-by-vendordata2.example.org."}

This seems like the right behavior to me but I'm open to suggestions if you think that the <name> value in nova.conf is intended to have a different meaning.

-Andrew

Revision history for this message
Manish Mahalwal (mmahalwa) wrote :

Thank you for your response @andrewboggot!

1. Can you confirm if the /var/lib/cloud/instances/<your-instance>/vendor-data2.txt contains the content of the YAML you passed?

2. When I set the name as cloud-init@<URL> in nova.conf, my custom REST service (to which Nova sends a POST request to) has to respond with a JSON string. The JSON string with which it responds is:
{"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}

The openstack/latest/vendor_data2.json file becomes:
{"cloud-init": {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}}

This too is not parsed by cloud-init. If the REST service responds with only the YAML (the value pair) which is `#cloud-config\npackage_upgrade: True\npackages:\n - htop`, Nova refuses to accept this string because it is expecting a VALID JSON. So, I am unable to remove that extra layer in my JSON, to make it exactly like your JSON. I also tried leaving the `name@` field empty, but that doesn't work too.

>>So, how do I remove that extra layer of JSON to test if this works as expected?

On a different note, whether <name> value has a different meaning:
I may be wrong on this but, the intention of having the `name` field is just to have a name for the REST service from which the JSON came. For example, I have several REST services to query from which I have to fetch vendor data.

From (https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/vendordata-reboot.html)
"Each REST microservice is configured by appending a new item to the list provided in the vendordata_dynamic_targets flag. This flag is composed of entries of the form:"

vendordata_dynamic_targets=name1@http://example.com,name2@http://example2.com

The name element for these microservices becomes the new key for the vendor_data2.json file like,

{
    "name1": {
        "cloud-init": "#cloud-config\n..."
    },
    "name2": {
        "cloud-init": "#cloud-config\n..."
    }
}

Please note I have taken the above example from the Nova specs mentioned earlier and extended it for 2 REST services. Now, if we consider your approach of having cloud-init as the outer key, then how are we going to differentiate b/w several REST services.

Revision history for this message
James Falcon (falcojr) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.