UnicodeDecodeError during mojo run

[Benjamin Allot]

Upon doing a new deployment, I ran in those errors:

$mojo run -m manifests/upgrade-newpatch
.
.
.
2020-11-02 15:42:48 [INFO] Waiting up to 1800 seconds for environment to become ready (not blocked or in maintenance)
2020-11-02 15:42:50 [ERROR] Unknown error
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/mojo/cli.py", line 684, in run_with_args
    args.func(args)
  File "/usr/lib/python3/dist-packages/mojo/utils.py", line 380, in wrapped
    return method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/mojo/cli.py", line 352, in run_from_manifest
    manifest.run(project, workspace, args.stage, args.interactive)
  File "/usr/lib/python3/dist-packages/mojo/manifest.py", line 126, in run
    phase_name.run(project, workspace, stage)
  File "/usr/lib/python3/dist-packages/mojo/phase.py", line 1094, in run
    timeout=status_timeout, wait_for_steady=True, max_wait=max_wait, wait_for_workload=wait_for_workload
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 495, in check_and_wait
    self.status(force_update=True)
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 587, in status
    self._raw_status = check_output_with_timeout(cmd, self._command_timeout, JujuStatusCommandTimedOutError)
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 44, in check_output_with_timeout
    return subprocess.check_output(cmd, universal_newlines=True)
  File "/usr/lib/python3.4/subprocess.py", line 609, in check_output
    output, unused_err = process.communicate(inputdata, timeout=timeout)
  File "/usr/lib/python3.4/subprocess.py", line 947, in communicate
    stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python3.4/subprocess.py", line 491, in _eintr_retry_call
    return func(*args)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 27878: ordinal not in range(128)

It seems to happen on every `mojo run`

$ mojo run -m manifests/deploy
.
.
.
2020-11-02 16:05:41 [INFO] Checking Juju status (timeout=1800)
2020-11-02 16:05:41 [INFO] Waiting up to 1800 seconds for environment to become ready (not blocked or in maintenance)
2020-11-02 16:05:42 [ERROR] Unknown error
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/mojo/cli.py", line 684, in run_with_args
    args.func(args)
  File "/usr/lib/python3/dist-packages/mojo/utils.py", line 380, in wrapped
    return method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/mojo/cli.py", line 352, in run_from_manifest
    manifest.run(project, workspace, args.stage, args.interactive)
  File "/usr/lib/python3/dist-packages/mojo/manifest.py", line 126, in run
    phase_name.run(project, workspace, stage)
  File "/usr/lib/python3/dist-packages/mojo/phase.py", line 891, in run
    juju_status.check_and_wait(timeout=status_timeout, wait_for_steady=wait_flag, max_wait=max_wait)
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 495, in check_and_wait
    self.status(force_update=True)
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 587, in status
    self._raw_status = check_output_with_timeout(cmd, self._command_timeout, JujuStatusCommandTimedOutError)
  File "/usr/lib/python3/dist-packages/mojo/juju/status.py", line 44, in check_output_with_timeout
    return subprocess.check_output(cmd, universal_newlines=True)
  File "/usr/lib/python3.4/subprocess.py", line 609, in check_output
    output, unused_err = process.communicate(inputdata, timeout=timeout)
  File "/usr/lib/python3.4/subprocess.py", line 947, in communicate
    stdout = _eintr_retry_call(self.stdout.read)
  File "/usr/lib/python3.4/subprocess.py", line 491, in _eintr_retry_call
    return func(*args)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 27878: ordinal not in range(128)

[Tom Haddon (mthaddon)]

Ok, so I've finally figured out what's going on here.

We don't pass through any environment variables to Mojo when executing this, and when using universal_newlines=True text is streamed using the encoding returned by locale.getpreferredencoding(False). Because we've not passed through (for example LANG) we end up with a preferred encoding of ANSI_X3.4-1968, which then fails on any unicode characters.

I was able to confirm the following fixes things (will discuss with a few folks whether this is the right approach and propose an MP):

=== modified file 'mojo/juju/status.py'
--- old/mojo/juju/status.py 2021-02-05 12:12:08 +0000
+++ new/mojo/juju/status.py 2021-04-15 09:47:31 +0000
@@ -1,5 +1,6 @@
 from __future__ import division, print_function
 import datetime
+import locale
 import logging
 import os
 import subprocess
@@ -41,6 +42,7 @@

     cmd.extend(command)
     try:
+ locale.setlocale(locale.LC_ALL, os.environ.get('LANG', 'en_US.UTF-8'))
         return subprocess.check_output(cmd, universal_newlines=True)
     except subprocess.CalledProcessError as cpe:
         if command_timeout is not None: