An "empty map" (which parses as Null) in an overlay can accidentally delete an entire charm or all of it's options

[Trent Lloyd]

If you have an overlay where one of the keys which should be a map (such as the 'options' section of an application, or, the application itself under 'applications') is left empty, it has the effect of deleting that entire section from the bundle.

Technically, we can understand why this is the case. Becuase YAML parsing is fairly implicit, these values are parsed not as a map but instead as empty/none/null. When merging the bundles, instead of merging the two maps it instead replaces the map with the empty value from the overlay, in the same way you might replace a string with another string. I assume (though have not checked) that the YAML merging basically replaces values in most cases unless it's an map in which case it merges them - possibly with some other special logic.

However this is surprising behaviour and likely to catch users off-guard and deploy a substantially incorrect bundle. I hit this because I commented out the only option set in an overlay bundle for nova-compute, which resulted in it deploying the wrong version of the software because the openstack-origin value was reset to the default 'distro' (along with all the other options which were set, though I have removed from the example here)

This kind of implicit parsing confusion exists in other places of YAML and is a common criticism of YAML. Another famous example is entries like "country: AU" parse as the string "AU" but "country: NO" will parse as a boolean False.

While I don't suggest we should solve every case I think this behaviour is so surprising and confusing and could be solved with some special casing. Possible suggestions

(1) Detect when an overlay sets a value that is expected to be, or already is, a map and throw a warning or error.

(2) Assume empty values to be an empty list, such as suggested in this similar issue for the Salt project:

https://github.com/saltstack/salt/issues/16069

(3) Parse literal empty values as an empty map but not explicit ~/!!null entries.

It's notable that users may in some cases actaually want to delete a previously set map? And these options may prevent that. But I do not feel like this is a very common likely cases versus an accidentally empty section.

I think this is maybe discussed here as a difference between yaml.v2 and yaml.v3 and similar ins/outs to the above. I'm not sure what the right answer is but my main request is "find a way to make it less likely a user does the wrong thing by accident".

https://github.com/go-yaml/yaml/issues/681

= Example =

[base.yaml]

applications:
  nova-compute:
    charm: ch:nova-compute
    channel: wallaby/edge
    options:
      openstack-origin: cloud:focal-wallaby

[overlay1.yaml]

applications:
  nova-compute:
    options:

# Result: The application is deployed but all of the options are deleted, so openstack-origin is set to the default value

[overlay2.yaml]

applications:
  nova-compute:

# Result: The application is not deployed, as the entire charm was deleted

[Trent Lloyd (lathiat)]

I originally hit this because I had commented out the only option in an overlay

ubuntu@lathiat-bastion:~/stsstack-bundles/openstack$ cat ../overlays/vault-openstack-secrets.yaml
applications:
  nova-compute:
    options:
# encrypt: True
    storage:
      ephemeral-device: cinder,32G,1
relations:
  - ['nova-compute:secrets-storage', 'vault:secrets']

[Trent Lloyd (lathiat)]

There is apparently at least one place the null technique is intentionally used by someone here to delete kubeapi-load-balancer:
https://ubuntu.com/kubernetes/docs/openstack-integration#api-server-load-balancer

They do explicitly write "null" though instead of just leaving it empty

[Harry Pidcock (hpidcock)]

I think the best option here is to make a breaking change in 3.1 or 3.2 to introduce safer default behaviour, where the defaulting of fields is more explicit.

[Vitaly Antonenko (anvial)]

Not sure we are ready to make this changes for 3.1, I'll change the milestone to 3.2