A Chinese character in the snapcraft.yaml crashes the snapcraft

[XiaoGuo, Liu]

In my snapcraft.yaml's summay, it contains an unrecoginized unicode space (between Piglow and API). When I try to use snapcraft to compile my project, it crashes:

name: piglow2
version: 1.0
vendor: XiaoGuo, Liu <email address hidden>
summary: Piglow　API
description: This is the webserver API to control the piglow
icon: icon.png

services:
  piglow2:
    start: bin/piglow2

parts:
  piglow:
    plugin: go
    source: ./src/piglow2

The output is like:

ubuntu@a5ceb8ea0825:~/work/piglow2$ snapcraft
Traceback (most recent call last):
  File "/usr/local/bin/snapcraft", line 4, in <module>
    __import__('pkg_resources').run_script('snapcraft==0.5', 'snapcraft')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 735, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1652, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/EGG-INFO/scripts/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/main.py", line 149, in main
    args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 103, in assemble
    snap(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 85, in snap
    cmd(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 346, in cmd
    config = _load_config()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 412, in _load_config
    _config = snapcraft.yaml.Config()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/yaml.py", line 97, in __init__
    self.data = _snapcraft_yaml_load()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]

Attached please find the snapcraft.yaml, and my project is at:

https://github.com/liu-xiao-guo/piglow2

[Sergio Schvezov (sergiusens)]

Can I get more details about your environment? I don't see this

sergiusens@lindon:~/source/github.com/liu-xiao-guo$ git clone https://github.com/liu-xiao-guo/piglow2
Clonar en «piglow2»...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 16 (delta 3), reused 16 (delta 3), pack-reused 0
Unpacking objects: 100% (16/16), done.
Checking connectivity... hecho.
sergiusens@lindon:~/source/github.com/liu-xiao-guo/piglow2$ ~/source/github.com/ubuntu-core/snapcraft/bin/snapcraft
Pulling piglow
env GOPATH=/home/sergiusens/source/github.com/liu-xiao-guo/piglow2/parts/piglow/go go get -t -d ./piglow2/...
Building piglow
env GOPATH=/home/sergiusens/source/github.com/liu-xiao-guo/piglow2/parts/piglow/go go install ./piglow2/...
Staging piglow
Snapping piglow
Snapping |
Errors
------
 - snappy-systemd:package_yaml_description_present:piglow2
 required description field not specified
 - snappy-systemd:package_yaml_required_key:piglow2:description
 Missing required field 'description'
piglow2_1.0_amd64.snap: FAIL
Snap «piglow2_1.0_amd64.snap» generado

[XiaoGuo, Liu (liu-xiao-guo)]

Sorry, this works on Ubuntu Desktop 15.04. However, the error happens when I compile it on an ARM device like raspberry 2 device. I will send you the document about my setup on the Raspberry PI device (I remember I shared it with you already). I compiled it in a container inside the docker.

I just did it again, and it is reproducible.

Thanks & best regards,
XiaoGuo

ubuntu@a5ceb8ea0825:~/work/piglow2$ ls
icon.png snapcraft.yaml src
ubuntu@a5ceb8ea0825:~/work/piglow2$ ../snapcraft/bin/snapcraft
Traceback (most recent call last):
  File "../snapcraft/bin/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/home/ubuntu/work/snapcraft/snapcraft/main.py", line 149, in main
    args.func(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 103, in assemble
    snap(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 85, in snap
    cmd(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 346, in cmd
    config = _load_config()
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 412, in _load_config
    _config = snapcraft.yaml.Config()
  File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 97, in __init__
    self.data = _snapcraft_yaml_load()
  File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 91: ordinal not in range(128)

[John Lenton (chipaca)]

could you attach the output of `printenv` (on the raspberry) to the bug please?

On 25 November 2015 at 10:12, XiaoGuo, Liu <email address hidden> wrote:
> Sorry, this works on Ubuntu Desktop 15.04. However, the error happens
> when I compile it on an ARM device like raspberry 2 device. I will send
> you the document about my setup on the Raspberry PI device (I remember I
> shared it with you already). I compiled it in a container inside the
> docker.
>
> I just did it again, and it is reproducible.
>
> Thanks & best regards,
> XiaoGuo
>
> ubuntu@a5ceb8ea0825:~/work/piglow2$ ls
> icon.png snapcraft.yaml src
> ubuntu@a5ceb8ea0825:~/work/piglow2$ ../snapcraft/bin/snapcraft
> Traceback (most recent call last):
> File "../snapcraft/bin/snapcraft", line 33, in <module>
> snapcraft.main.main()
> File "/home/ubuntu/work/snapcraft/snapcraft/main.py", line 149, in main
> args.func(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 103, in assemble
> snap(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 85, in snap
> cmd(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 346, in cmd
> config = _load_config()
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 412, in _load_config
> _config = snapcraft.yaml.Config()
> File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 97, in __init__
> self.data = _snapcraft_yaml_load()
> File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
> return yaml.load(fp)
> File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
> loader = Loader(stream)
> File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
> Reader.__init__(self, stream)
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
> self.determine_encoding()
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
> self.update_raw()
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
> data = self.stream.read(size)
> File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 91: ordinal not in range(128)
>
> --
> You received this bug notification because you are a member of Snappy
> Developers, which is subscribed to Snapcraft.
> https://bugs.launchpad.net/bugs/1518150
>
> Title:
> A Chinese character in the snapcraft.yaml crashes the snapcraft
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapcraft/+bug/1518150/+subscriptions

[XiaoGuo, Liu (liu-xiao-guo)]

Hi John,

Below is the container info on the raspberry pi device. The container runs inside my docker. I have just shared my setup document with you. Please check your email.

ubuntu@a5ceb8ea0825:~/work$ printenv
HOSTNAME=a5ceb8ea0825
TERM=xterm
PULSE_SERVER=tcp:172.17.42.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/ubuntu/work
PWD=/home/ubuntu/work
SHLVL=1
HOME=/home/ubuntu
GOROOT=/usr/local/go
GOPATH=/home/ubuntu/work
DISPLAY=:0
OLDPWD=/
_=/usr/bin/printenv

This the environment set up on my Raspberry PI device.

(RaspberryPi2)ubuntu@localhost:~$ snappy list -v
Name Date Version Developer
ubuntu-core 2015-09-25 2 ubuntu*
ubuntu-core 2015-11-13 3 ubuntu!
config-example 2015-11-25 1.0.6 canonical*
docker 2015-11-09 1.6.2.004 canonical
docker 2015-11-13 1.6.2.005 sideload*
hello-world 2015-11-12 1.0.18 canonical*
mqtt-piglow 2015-11-23 IGJWYKeGLWea sideload*
mqtttestclient 2015-11-12 IFOTYdKLCaGB sideload*
piglow 2015-11-20 IGCNEEceALce sideload*
snappy-debug 2015-11-13 0.5 canonical
snappy-debug 2015-11-14 0.6 canonical*
webcam-demo 2015-11-12 1.0.2 canonical*
webcam-webui 2015-11-17 IFbYNNbSUdHS sideload*
webcam 2015-11-18 IFeGRLEgffcQ sideload*
webdm 2015-10-27 0.9.3 sideload
webdm 2015-10-29 0.9.4 sideload*
pi2 2015-09-25 0.16 canonical*
Reboot to use the new ubuntu-core.

[XiaoGuo, Liu (liu-xiao-guo)]

Attached please find the captured content of the snapcraft.yaml file inside the container.

[XiaoGuo, Liu (liu-xiao-guo)]

Attached please find the captured content of the snapcraft.yaml file inside the container.

[Sergio Schvezov (sergiusens)]

With a code point U+20021

We get
Traceback (most recent call last):
  File "/usr/bin/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/usr/lib/python3/dist-packages/snapcraft/main.py", line 148, in main
    args.func(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 98, in assemble
    snap(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 79, in snap
    cmd(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 307, in cmd
    config = _load_config()
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 373, in _load_config
    _config = snapcraft.yaml.Config()
  File "/usr/lib/python3/dist-packages/snapcraft/yaml.py", line 98, in __init__
    self.data = _snapcraft_yaml_load()
  File "/usr/lib/python3/dist-packages/snapcraft/yaml.py", line 324, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 135, in determine_encoding
    self.update(1)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 169, in update
    self.check_printable(data)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 144, in check_printable
    'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #x20021: special characters are not allowed
  in "snapcraft.yaml", position 96

[John Lenton (chipaca)]

I'm not sure why it seems to think anything not in the first plane is a "special character". The yaml spec says
"The allowed character range explicitly excludes the surrogate block #xD800-#xDFFF, DEL #x7F, the C0 control block #x0-#x1F (except for #x9, #xA, and #xD), the C1 control block #x80-#x9F, #xFFFE, and #xFFFF."

I've tried a few characters that need 3 or 4 bytes in utf8 (because I was looking at bugs in utf16 in a yaml parser in a different language) and found that pyyaml aborts with them. At first I thought it was confusing the utf8 encoding with the codepoint, but U+20021 (𠀡) is not that.