consider setting proper locale for hooks environments

Bug #1334482 reported by Jesse Meek
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Medium
Tim Penhey
MongoDB Charm
Won't Fix
Low
Unassigned

Bug Description

A mongodb-relation-changed hook called out to a python3 script to import some data into mongodb.

The python script encountered a unicode error:

2014-06-19 20:23:17 INFO mongodb-relation-changed Traceback (most recent call last):
2014-06-19 20:23:17 INFO mongodb-relation-changed File "/opt/openmotion/import.py", line 194, in
2014-06-19 20:23:17 INFO mongodb-relation-changed parse_bikes(mongo_uri)
2014-06-19 20:23:17 INFO mongodb-relation-changed File "/opt/openmotion/import.py", line 155, in parse_bikes
2014-06-19 20:23:17 INFO mongodb-relation-changed stations = parser1
2014-06-19 20:23:17 INFO mongodb-relation-changed File "/opt/openmotion/import.py", line 81, in parse_valencia_bikes
2014-06-19 20:23:17 INFO mongodb-relation-changed json_data = open(basepath + 'data/bikes/Valenbisi.JSON').read()
2014-06-19 20:23:17 INFO mongodb-relation-changed File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
2014-06-19 20:23:17 INFO mongodb-relation-changed return codecs.ascii_decode(input, self.errors)[0]
2014-06-19 20:23:17 INFO mongodb-relation-changed UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 272: ordinal not in range(128)
2014-06-19 20:23:17 ERROR juju.worker.uniter uniter.go:482 hook failed: exit status 1

Running the same script from a terminal on the same node succeeded.

After further debugging, I determined that the mongodb-relation-changed hook does not set any locale.

Adding:

export LANG=en_US.UTF-8

to the hook allowed the import script to succeed.

Note: although this enabled me to make progress on writing my hook, I think upstream needs to take a closer look. I am not an expert on locales, and I wonder if forcing en_US.UTF-8 on all juju users is appropriate or not...

Copied from github (also see discussion there): https://github.com/juju/juju/issues/133

Revision history for this message
Stuart Bishop (stub) wrote :

The traceback is showing a bug in the mongodb charm:

2014-06-19 20:23:17 INFO mongodb-relation-changed json_data = open(basepath + 'data/bikes/Valenbisi.JSON').read()

It is attempting to open a file in text mode without specifying the encoding, which will fail in some environments. You could force the system locale, which would mean this bug would only be triggered in locales not using UTF-8 encoding. Forcing a specific locale is not really appropriate, as if you are not using the system locale or you will end up with much harder to solve problems (things being sorted in the wrong order, unreadable filenames).

As the preferred encoding of JSON is UTF-8, the fix is to specify this in the open call (if you want read() to return unicode text), or open the file in binary mode if you want a UTF-8 encoded byte string.

Jesse Meek (waigani)
Changed in juju-core:
assignee: nobody → Jesse Meek (waigani)
Jesse Meek (waigani)
Changed in juju-core:
assignee: Jesse Meek (waigani) → nobody
Revision history for this message
Cory Benfield (cory-benfield) wrote :

I stumbled into this while working on charms for Project Calico. Adding a PPA as part of a hook's install process is fine, but if that PPA's public key name has non-ASCII characters in it a call to apt_update can fail and remove the GPG key for the PPA. That blocks downloading of the packages (and rightly so!), which blocks install.

Curtis Hovey (sinzui)
Changed in juju-core:
importance: High → Medium
Changed in juju-core:
milestone: none → 2.0.0
tags: added: mongodb
affects: juju-core → juju
Changed in juju:
milestone: 2.0.0 → none
milestone: none → 2.0.0
Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.0.0 → 2.0.1
Curtis Hovey (sinzui)
Changed in juju:
milestone: 2.0.1 → none
affects: mongodb (Juju Charms Collection) → mongodb-charm
Stuart Bishop (stub)
tags: added: canonical-is
Revision history for this message
Joel Sing (jsing) wrote :

Please export LANG=C.UTF-8 in the hook execution environment (this is available across trusty/xenial/bionic and should be similar on other platforms).

Revision history for this message
Tim Penhey (thumper) wrote :

@jsing 2.6.x or 2.7 for this? It seems innocuous, but it may impact things. Perhaps just 2.7?

Revision history for this message
Joel Sing (jsing) wrote :

@thumper given it has never been set before, I would go for 2.7 onwards.

Tim Penhey (thumper)
Changed in juju:
milestone: none → 2.7-beta1
assignee: nobody → Tim Penhey (thumper)
Revision history for this message
Tim Penhey (thumper) wrote :
Changed in juju:
status: Triaged → In Progress
Tim Penhey (thumper)
Changed in juju:
status: In Progress → Fix Committed
Changed in juju:
status: Fix Committed → Fix Released
Revision history for this message
Xav Paice (xavpaice) wrote :

To close this for the Mongo charm, we should specify UTF-8 in the open() call (for reads) in hooks.py.

Changed in charm-mongodb:
status: New → Triaged
importance: Undecided → Low
Eric Chen (eric-chen)
Changed in charm-mongodb:
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.