juju-deployer (maybe?) doing crazy things with include-file and include-base64 configurations

[Daniel Manrique]

Filed as https://portal.admin.canonical.com/C127313/ initially, but I crafted a smaller repro case and it looks like it's something to do with juju-deployer.

export MOJO_PROJECT=minispec
export MOJO_STAGE=devel
export MOJO_WORKSPACE=mojobug
export MOJO_SERIES=bionic

$ echo "Whatever, this will be borked by mojo I think." > vhost.template

$ cat << EOF > services
foo:
        series: bionic
        services:
                apache:
                        charm: cs:apache2
                        series: bionic
                        options:
                            vhost_template_vars: include-base64://{{ spec_dir }}/vhost.template
EOF

$ mojo deploy-show --options config=services
series: bionic
services:
  apache:
    charm: cs:apache2
    options:
      vhost_template_vars: !!binary |
        VjJoaGRHVjJaWElzSUhSb2FYTWdkMmxzYkNCaVpTQmliM0pyWldRZ1lua2diVzlxYnlCSklIUm9h
        VzVyTGdvPQ==
    series: bionic

Notice that vhost_template_vars is !!binary, which it shouldn't be.

I went poking around mojo itself, and its jinja2 template substitution correctly yields something like:

foo:
        series: bionic
        services:
                apache:
                        charm: cs:apache2
                        series: bionic
                        options:
                            vhost_template_vars: include-base64:///home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/spec/vhost.template

I then pass this file to deployer:

$ juju-deployer --diff -c /home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/charms/services foo
services:
  modified:
    apache:
      cfg-config:
        vhost_template_vars: !!binary |
          VjJoaGRHVjJaWElzSUhSb2FYTWdkMmxzYkNCaVpTQmliM0pyWldRZ1lua2diVzlxYnlCSklIUm9h
          VzVyTGdvPQ==
      env-config:
        vhost_template_vars: b'V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo='

so AHA< it *is* deployer munging those .template files.

Funny thing is, deployer does not seem to have changed since March...

Mojo does this using deployer code directly:

        deployer_config = ConfigStack(configs)
        deployer_config.load()
        deployment = deployer_config.get(deploy_name)
        deployment.repo_path = workspace.repo_dir
        deployment.resolve_config()

This small patch to deployer (it's entirely incomplete) improves things a bit:

=== modified file 'deployer/deployment.py'
--- deployer/deployment.py 2017-03-16 04:23:06 +0000
+++ deployer/deployment.py 2020-08-20 21:42:16 +0000
@@ -270,7 +270,7 @@
                 v = fh.read()
                 if include_type == "base64":
                     v = b64encode(v)
- return v
+ return v.encode('utf-8')
         if feedback:
             return feedback

$ juju-deployer --diff -c /home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/charms/services foo
services:
  modified:
    apache:
      cfg-config:
        vhost_template_vars: V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo=
      env-config:
        vhost_template_vars: b'V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo='

(presumably a similar transform is needed for env-config).

Noticing that the package Mojo depends on is python3-juju-deployer, this smells like deployer was switched to python3 without checking for all str/bytes implicit conversions, the above section likely being one of them - which is why this code is returning bytes/binary data where earlier it should have returned strings.

[Daniel Manrique (roadmr)]

An implication of the fact that this is a juju-deployer issue might be that bundle phases will be unaffected by this problem :)

[James Simpson (jsimpso)]

We seem to have hit this bug today while deploying a mojo manifest containing:

haproxy:
  options:
    services: include-file://{{spec_dir}}/{{stage}}/../configs/haproxy.cfg

"haproxy.cfg" being a file containing valid yaml data.

Instead of the expected result (haproxy.cfg being parsed and the values correctly propagating `juju config haproxy services`), what we saw was the entire contents of that file being wrapped as a binary string, e.g.:

$ juju config haproxy services
b'- service_name: service\n service_host: "0.0.0.0"\n service_port: 443\n service_options:\n - mode http\n - balance leastconn\n - cookie SRVNAME insert\n'

This bug does not appear to occur for another spec that includes haproxy configuration in the same manner, but does it in a bundle phase instead of a deploy phase.

[Daniel Manrique (roadmr)]

@James, that makes sense per my findings; bundle phases are handled natively by Juju (deployed as a bundle), whereas deploy phases use juju-deployer which has the bug (this is a remnant from when Juju had no native bundle support).

A valid workaround, if your spec allows it, is to port it to use only bundle phases, thus bypassing juju-deployer entirely.

[Loïc Gomez (kotodama)]

It seems we it that bug on a deployment today too (mailman3-web):

$ mojo deploy-diff --preview-config
/srv/juju-dist/latest-2.x/usr/bin/juju config mailman3-web extra_packages=django-mailman-launchpad-teams python-contextlib2 python-django-openid-auth python-raven check-lists-conf=b'W2....

mojo deply-show shows a few !!binary values.

[Tom Haddon (mthaddon)]

I believe this is now fixed per https://pastebin.ubuntu.com/p/2SP5mTjrYM/.