juju-deployer (maybe?) doing crazy things with include-file and include-base64 configurations

Bug #1892423 reported by Daniel Manrique on 2020-08-20
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Mojo: Continuous Delivery for Juju
Low
Unassigned
juju-deployer
Critical
Unassigned

Bug Description

Filed as https://portal.admin.canonical.com/C127313/ initially, but I crafted a smaller repro case and it looks like it's something to do with juju-deployer.

export MOJO_PROJECT=minispec
export MOJO_STAGE=devel
export MOJO_WORKSPACE=mojobug
export MOJO_SERIES=bionic

$ echo "Whatever, this will be borked by mojo I think." > vhost.template

$ cat << EOF > services
foo:
        series: bionic
        services:
                apache:
                        charm: cs:apache2
                        series: bionic
                        options:
                            vhost_template_vars: include-base64://{{ spec_dir }}/vhost.template
EOF

$ mojo deploy-show --options config=services
series: bionic
services:
  apache:
    charm: cs:apache2
    options:
      vhost_template_vars: !!binary |
        VjJoaGRHVjJaWElzSUhSb2FYTWdkMmxzYkNCaVpTQmliM0pyWldRZ1lua2diVzlxYnlCSklIUm9h
        VzVyTGdvPQ==
    series: bionic

Notice that vhost_template_vars is !!binary, which it shouldn't be.

I went poking around mojo itself, and its jinja2 template substitution correctly yields something like:

foo:
        series: bionic
        services:
                apache:
                        charm: cs:apache2
                        series: bionic
                        options:
                            vhost_template_vars: include-base64:///home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/spec/vhost.template

I then pass this file to deployer:

$ juju-deployer --diff -c /home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/charms/services foo
services:
  modified:
    apache:
      cfg-config:
        vhost_template_vars: !!binary |
          VjJoaGRHVjJaWElzSUhSb2FYTWdkMmxzYkNCaVpTQmliM0pyWldRZ1lua2diVzlxYnlCSklIUm9h
          VzVyTGdvPQ==
      env-config:
        vhost_template_vars: b'V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo='

so AHA< it *is* deployer munging those .template files.

Funny thing is, deployer does not seem to have changed since March...

Mojo does this using deployer code directly:

        deployer_config = ConfigStack(configs)
        deployer_config.load()
        deployment = deployer_config.get(deploy_name)
        deployment.repo_path = workspace.repo_dir
        deployment.resolve_config()

This small patch to deployer (it's entirely incomplete) improves things a bit:

=== modified file 'deployer/deployment.py'
--- deployer/deployment.py 2017-03-16 04:23:06 +0000
+++ deployer/deployment.py 2020-08-20 21:42:16 +0000
@@ -270,7 +270,7 @@
                 v = fh.read()
                 if include_type == "base64":
                     v = b64encode(v)
- return v
+ return v.encode('utf-8')
         if feedback:
             return feedback

$ juju-deployer --diff -c /home/ubuntu/.local/share/mojo/minispec/bionic/mojobug/charms/services foo
services:
  modified:
    apache:
      cfg-config:
        vhost_template_vars: V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo=
      env-config:
        vhost_template_vars: b'V2hhdGV2ZXIsIHRoaXMgd2lsbCBiZSBib3JrZWQgYnkgbW9qbyBJIHRoaW5rLgo='

(presumably a similar transform is needed for env-config).

Noticing that the package Mojo depends on is python3-juju-deployer, this smells like deployer was switched to python3 without checking for all str/bytes implicit conversions, the above section likely being one of them - which is why this code is returning bytes/binary data where earlier it should have returned strings.

Daniel Manrique (roadmr) wrote :

An implication of the fact that this is a juju-deployer issue might be that bundle phases will be unaffected by this problem :)

Barry Price (barryprice) on 2020-08-21
Changed in mojo:
status: New → Triaged
importance: Undecided → Critical
Barry Price (barryprice) on 2020-08-27
Changed in juju-deployer:
status: New → Triaged
importance: Undecided → Critical
Changed in mojo:
status: Triaged → Invalid
importance: Critical → Undecided
Tom Haddon (mthaddon) on 2020-10-27
Changed in mojo:
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers