A Chinese character in the snapcraft.yaml crashes the snapcraft

Bug #1518150 reported by XiaoGuo, Liu on 2015-11-20
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Snapcraft
High
Sergio Schvezov
pyyaml (Debian)
Fix Released
Unknown
pyyaml (Ubuntu)
High
Barry Warsaw

Bug Description

In my snapcraft.yaml's summay, it contains an unrecoginized unicode space (between Piglow and API). When I try to use snapcraft to compile my project, it crashes:

name: piglow2
version: 1.0
vendor: XiaoGuo, Liu <email address hidden>
summary: Piglow API
description: This is the webserver API to control the piglow
icon: icon.png

services:
  piglow2:
    start: bin/piglow2

parts:
  piglow:
    plugin: go
    source: ./src/piglow2

The output is like:

ubuntu@a5ceb8ea0825:~/work/piglow2$ snapcraft
Traceback (most recent call last):
  File "/usr/local/bin/snapcraft", line 4, in <module>
    __import__('pkg_resources').run_script('snapcraft==0.5', 'snapcraft')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 735, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1652, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/EGG-INFO/scripts/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/main.py", line 149, in main
    args.func(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 103, in assemble
    snap(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 85, in snap
    cmd(args)
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 346, in cmd
    config = _load_config()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/cmds.py", line 412, in _load_config
    _config = snapcraft.yaml.Config()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/yaml.py", line 97, in __init__
    self.data = _snapcraft_yaml_load()
  File "/usr/local/lib/python3.4/dist-packages/snapcraft-0.5-py3.4.egg/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]

Attached please find the snapcraft.yaml, and my project is at:

https://github.com/liu-xiao-guo/piglow2

XiaoGuo, Liu (liu-xiao-guo) wrote :
Sergio Schvezov (sergiusens) wrote :

Can I get more details about your environment? I don't see this

sergiusens@lindon:~/source/github.com/liu-xiao-guo$ git clone https://github.com/liu-xiao-guo/piglow2
Clonar en «piglow2»...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 16 (delta 3), reused 16 (delta 3), pack-reused 0
Unpacking objects: 100% (16/16), done.
Checking connectivity... hecho.
sergiusens@lindon:~/source/github.com/liu-xiao-guo/piglow2$ ~/source/github.com/ubuntu-core/snapcraft/bin/snapcraft
Pulling piglow
env GOPATH=/home/sergiusens/source/github.com/liu-xiao-guo/piglow2/parts/piglow/go go get -t -d ./piglow2/...
Building piglow
env GOPATH=/home/sergiusens/source/github.com/liu-xiao-guo/piglow2/parts/piglow/go go install ./piglow2/...
Staging piglow
Snapping piglow
Snapping |
Errors
------
 - snappy-systemd:package_yaml_description_present:piglow2
 required description field not specified
 - snappy-systemd:package_yaml_required_key:piglow2:description
 Missing required field 'description'
piglow2_1.0_amd64.snap: FAIL
Snap «piglow2_1.0_amd64.snap» generado

Changed in snapcraft:
status: New → Incomplete
XiaoGuo, Liu (liu-xiao-guo) wrote :

Sorry, this works on Ubuntu Desktop 15.04. However, the error happens when I compile it on an ARM device like raspberry 2 device. I will send you the document about my setup on the Raspberry PI device (I remember I shared it with you already). I compiled it in a container inside the docker.

I just did it again, and it is reproducible.

Thanks & best regards,
XiaoGuo

ubuntu@a5ceb8ea0825:~/work/piglow2$ ls
icon.png snapcraft.yaml src
ubuntu@a5ceb8ea0825:~/work/piglow2$ ../snapcraft/bin/snapcraft
Traceback (most recent call last):
  File "../snapcraft/bin/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/home/ubuntu/work/snapcraft/snapcraft/main.py", line 149, in main
    args.func(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 103, in assemble
    snap(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 85, in snap
    cmd(args)
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 346, in cmd
    config = _load_config()
  File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 412, in _load_config
    _config = snapcraft.yaml.Config()
  File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 97, in __init__
    self.data = _snapcraft_yaml_load()
  File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 91: ordinal not in range(128)

could you attach the output of `printenv` (on the raspberry) to the bug please?

On 25 November 2015 at 10:12, XiaoGuo, Liu <email address hidden> wrote:
> Sorry, this works on Ubuntu Desktop 15.04. However, the error happens
> when I compile it on an ARM device like raspberry 2 device. I will send
> you the document about my setup on the Raspberry PI device (I remember I
> shared it with you already). I compiled it in a container inside the
> docker.
>
> I just did it again, and it is reproducible.
>
> Thanks & best regards,
> XiaoGuo
>
> ubuntu@a5ceb8ea0825:~/work/piglow2$ ls
> icon.png snapcraft.yaml src
> ubuntu@a5ceb8ea0825:~/work/piglow2$ ../snapcraft/bin/snapcraft
> Traceback (most recent call last):
> File "../snapcraft/bin/snapcraft", line 33, in <module>
> snapcraft.main.main()
> File "/home/ubuntu/work/snapcraft/snapcraft/main.py", line 149, in main
> args.func(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 103, in assemble
> snap(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 85, in snap
> cmd(args)
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 346, in cmd
> config = _load_config()
> File "/home/ubuntu/work/snapcraft/snapcraft/cmds.py", line 412, in _load_config
> _config = snapcraft.yaml.Config()
> File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 97, in __init__
> self.data = _snapcraft_yaml_load()
> File "/home/ubuntu/work/snapcraft/snapcraft/yaml.py", line 312, in _snapcraft_yaml_load
> return yaml.load(fp)
> File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
> loader = Loader(stream)
> File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
> Reader.__init__(self, stream)
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
> self.determine_encoding()
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 124, in determine_encoding
> self.update_raw()
> File "/usr/lib/python3/dist-packages/yaml/reader.py", line 178, in update_raw
> data = self.stream.read(size)
> File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 91: ordinal not in range(128)
>
> --
> You received this bug notification because you are a member of Snappy
> Developers, which is subscribed to Snapcraft.
> https://bugs.launchpad.net/bugs/1518150
>
> Title:
> A Chinese character in the snapcraft.yaml crashes the snapcraft
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapcraft/+bug/1518150/+subscriptions

XiaoGuo, Liu (liu-xiao-guo) wrote :

Hi John,

Below is the container info on the raspberry pi device. The container runs inside my docker. I have just shared my setup document with you. Please check your email.

ubuntu@a5ceb8ea0825:~/work$ printenv
HOSTNAME=a5ceb8ea0825
TERM=xterm
PULSE_SERVER=tcp:172.17.42.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/home/ubuntu/work
PWD=/home/ubuntu/work
SHLVL=1
HOME=/home/ubuntu
GOROOT=/usr/local/go
GOPATH=/home/ubuntu/work
DISPLAY=:0
OLDPWD=/
_=/usr/bin/printenv

This the environment set up on my Raspberry PI device.

(RaspberryPi2)ubuntu@localhost:~$ snappy list -v
Name Date Version Developer
ubuntu-core 2015-09-25 2 ubuntu*
ubuntu-core 2015-11-13 3 ubuntu!
config-example 2015-11-25 1.0.6 canonical*
docker 2015-11-09 1.6.2.004 canonical
docker 2015-11-13 1.6.2.005 sideload*
hello-world 2015-11-12 1.0.18 canonical*
mqtt-piglow 2015-11-23 IGJWYKeGLWea sideload*
mqtttestclient 2015-11-12 IFOTYdKLCaGB sideload*
piglow 2015-11-20 IGCNEEceALce sideload*
snappy-debug 2015-11-13 0.5 canonical
snappy-debug 2015-11-14 0.6 canonical*
webcam-demo 2015-11-12 1.0.2 canonical*
webcam-webui 2015-11-17 IFbYNNbSUdHS sideload*
webcam 2015-11-18 IFeGRLEgffcQ sideload*
webdm 2015-10-27 0.9.3 sideload
webdm 2015-10-29 0.9.4 sideload*
pi2 2015-09-25 0.16 canonical*
Reboot to use the new ubuntu-core.

XiaoGuo, Liu (liu-xiao-guo) wrote :

Attached please find the captured content of the snapcraft.yaml file inside the container.

XiaoGuo, Liu (liu-xiao-guo) wrote :

Attached please find the captured content of the snapcraft.yaml file inside the container.

Changed in snapcraft:
status: Incomplete → Fix Committed
importance: Undecided → High
assignee: nobody → Sergio Schvezov (sergiusens)
milestone: none → 0.6
Sergio Schvezov (sergiusens) wrote :

With a code point U+20021

We get
Traceback (most recent call last):
  File "/usr/bin/snapcraft", line 33, in <module>
    snapcraft.main.main()
  File "/usr/lib/python3/dist-packages/snapcraft/main.py", line 148, in main
    args.func(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 98, in assemble
    snap(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 79, in snap
    cmd(args)
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 307, in cmd
    config = _load_config()
  File "/usr/lib/python3/dist-packages/snapcraft/cmds.py", line 373, in _load_config
    _config = snapcraft.yaml.Config()
  File "/usr/lib/python3/dist-packages/snapcraft/yaml.py", line 98, in __init__
    self.data = _snapcraft_yaml_load()
  File "/usr/lib/python3/dist-packages/snapcraft/yaml.py", line 324, in _snapcraft_yaml_load
    return yaml.load(fp)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 70, in load
    loader = Loader(stream)
  File "/usr/lib/python3/dist-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 135, in determine_encoding
    self.update(1)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 169, in update
    self.check_printable(data)
  File "/usr/lib/python3/dist-packages/yaml/reader.py", line 144, in check_printable
    'unicode', "special characters are not allowed")
yaml.reader.ReaderError: unacceptable character #x20021: special characters are not allowed
  in "snapcraft.yaml", position 96

John Lenton (chipaca) wrote :

I'm not sure why it seems to think anything not in the first plane is a "special character". The yaml spec says
"The allowed character range explicitly excludes the surrogate block #xD800-#xDFFF, DEL #x7F, the C0 control block #x0-#x1F (except for #x9, #xA, and #xD), the C1 control block #x80-#x9F, #xFFFE, and #xFFFF."

I've tried a few characters that need 3 or 4 bytes in utf8 (because I was looking at bugs in utf16 in a yaml parser in a different language) and found that pyyaml aborts with them. At first I thought it was confusing the utf8 encoding with the codepoint, but U+20021 (𠀡) is not that.

Changed in snapcraft:
status: Fix Committed → Fix Released
Barry Warsaw (barry) on 2015-12-02
Changed in pyyaml (Ubuntu):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Barry Warsaw (barry)
Changed in pyyaml (Debian):
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.