mojo bundle-show doesn't properly merge/update config files

[Daniel Manrique]

(Quickly-filed bug due to impending EOD< feel free to mark incomplete, I plan to flesh it out and propose a proper patch with tests next week):

One of our specs uses two config files, one has stuff like:

templates:
  applications:
    sca-app:
      charm: {{charm_dir}}/software-center-agent
      num_units: 1
      constraints: mem=2G
      loadbalancer: sca-app-lb:reverseproxy
      options:
        active: false
        email_hostport: smtp.internal:25
        environment: {{ stage_name }}

the other has stuff like:

templates:
  applications:
    sca-app:
      options:
        metrics_target: ""
    sca-worker:
      options:
        metrics_target: ""

What we found is that, when this is converted to Python dicts and merged by Mojo's BundlePhase.show() logic, because it uses Python dictionary update() which is not recursive, the resulting dict looks like:

 'templates': {'applications': {'sca-app': {'options': {'metrics_target': ''}},
                                'sca-worker': {'options': {'metrics_target': ''}}}}}

So what's happening is each subsequent file's contents overwrite, rather than being merged, with the first-level values in the destination.

This patch to mojo makes config file/dict merging/updating sane by doing it recursively, note the recursive dict update function was shamelessly taken from stackoverflow.

=== modified file 'mojo/phase.py'
--- mojo/phase.py 2020-07-09 13:36:06 +0000
+++ mojo/phase.py 2020-07-10 20:09:20 +0000
@@ -25,6 +25,18 @@

 JUJU_EXE_PATH, _ = get_command("juju")

+# Python update don't suffice, must be cleverer than the wolf
+from collections.abc import Mapping
+def update(orig_dict, new_dict):
+ for key, val in new_dict.items():
+ if isinstance(val, Mapping):
+ tmp = update(orig_dict.get(key, { }), val)
+ orig_dict[key] = tmp
+ elif isinstance(val, list):
+ orig_dict[key] = (orig_dict.get(key, []) + val)
+ else:
+ orig_dict[key] = new_dict[key]
+ return orig_dict

 class NoSuchPhaseError(Exception):
     pass
@@ -1392,10 +1404,11 @@
         for config in configs:
             with open(config, 'r') as f:
                 config_yaml = yaml.safe_load(f.read())
- output.update(config_yaml)
+ # Use wolf update here
+ output = update(output, config_yaml)
         if return_output:
             return output
- print(output)
+ print(yaml.dump(output))

 class BundleDiff(BundlePhase):

[Tom Haddon (mthaddon)]

This has been fixed.