UTF-8 support in User Data (text/x-shellscript) is broken

Bug #1768600 reported by Alexey Matyushin on 2018-05-02
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cloud-init
Medium
Unassigned

Bug Description

Hi there,

(1),
Q. Tell us your cloud provider
A. AWS

(2),
Q. Any appropriate cloud-init configuration you can provide us
A. Default,

        $ cat /var/run/cloud-init/cloud.cfg
        di_report:
          datasource_list: [ Ec2, None ]

(3),
Q. Perform the following on the system and attach it to this bug:
A. Output of `cloud-init collect-logs` (filename: cloud-init.tar.gz) in attach.

(4),
Example of User Data used (can be reproduced with any UTF-8 character),

        #!/bin/bash
        # Basic comment with UTF-8 character - Ä
        echo "Repro"
        exit 0

(5),
Version of python3 installed (only python3 is installed (i.e. no python2 packages installed)),

        $ python3 -V
        Python 3.5.2

(6),
Version of cloud-init installed,

        $ cloud-init --version
        /usr/bin/cloud-init 17.2

Could you please have a look?

Thanks a lot in advance,
Alexey

Related branches

Alexey Matyushin (airgunster) wrote :
Alexey Matyushin (airgunster) wrote :

Also reproducible with boot commands as well,

`/var/lib/cloud/instances/i-0123456789abc/user-data.txt`
        #cloud-config

        # boot commands
        # default: none
        # this is very similar to runcmd, but commands run very early
        # in the boot process, only slightly after a 'boothook' would run.
        # bootcmd should really only be used for things that could not be
        # done later in the boot process. bootcmd is very much like
        # boothook, but possibly with more friendly.
        # - bootcmd will run on every boot
        # - the INSTANCE_ID variable will be set to the current instance id.
        # - you can use 'cloud-init-per' command to help only run once Ä
        bootcmd:
         - echo Reprod | tee -a /home/ubuntu/tee
         - echo Ä

$ sudo grep -iE 'err|warn|shells|v. 1' /var/log/cloud-init.log
        2018-05-03 12:00:55,484 - util.py[DEBUG]: Cloud-init v. 17.2 running 'init-local' at Thu, 03 May 2018 12:00:55 +0000. Up 6.24 seconds.
        2018-05-03 12:00:56,435 - DataSourceEc2.py[DEBUG]: strict_mode: warn, cloud_platform=AWS
        2018-05-03 12:00:56,526 - DataSourceEc2.py[DEBUG]: Found preferred metadata version 2016-09-02
        2018-05-03 12:00:58,401 - util.py[DEBUG]: Cloud-init v. 17.2 running 'init' at Thu, 03 May 2018 12:00:58 +0000. Up 9.23 seconds.
        2018-05-03 12:00:58,473 - stages.py[DEBUG]: Added default handler for {'text/x-shellscript'} from ShellScriptPartHandler: [['text/x-shellscript']]
        2018-05-03 12:00:58,473 - __init__.py[DEBUG]: Calling handler ShellScriptPartHandler: [['text/x-shellscript']] (__begin__, None, 2) with frequency once-per-instance
        2018-05-03 12:00:58,474 - util.py[WARNING]: Failed loading yaml blob
        yaml.reader.ReaderError: unacceptable character #xdcc4: special characters are not allowed
        2018-05-03 12:00:58,483 - util.py[WARNING]: Failed at merging in cloud config part from part-001
        AttributeError: 'NoneType' object has no attribute 'pop'
        2018-05-03 12:00:58,488 - __init__.py[DEBUG]: Calling handler ShellScriptPartHandler: [['text/x-shellscript']] (__end__, None, 2) with frequency once-per-instance
        2018-05-03 12:01:00,631 - util.py[DEBUG]: Cloud-init v. 17.2 running 'modules:config' at Thu, 03 May 2018 12:01:00 +0000. Up 11.43 seconds.
        2018-05-03 12:01:02,193 - util.py[DEBUG]: Cloud-init v. 17.2 running 'modules:final' at Thu, 03 May 2018 12:01:02 +0000. Up 12.99 seconds.
        2018-05-03 12:01:02,371 - util.py[DEBUG]: Cloud-init v. 17.2 finished at Thu, 03 May 2018 12:01:02 +0000. Datasource DataSourceEc2Local. Up 13.22 seconds

Logs are attached as well (filename: 2018-05-03T13-05_0100_boot_commands_cloud-init.tar.gz).

Chad Smith (chad.smith) wrote :

Thanks for this good bug filing against cloud-init 17.2 and for making cloud-init better.

I validated that the traceback is still seen in 18.2.4

steps to reproduce with lxc

cat > runcmd.yaml <<EOF
#cloud-config
runcmd:
 - echo Ä
EOF

lxc init ubuntu-daily:bionic myb1
lxc config set myb1 user.user-data - < runcmd.yaml
lxc start myb1;
lxc exec myb1 cloud-init status --wait --long;
lxc exec myb1 grep yaml.reader /var/log/cloud-init.log

Changed in cloud-init:
status: New → Confirmed
importance: Undecided → Medium
Alexey Matyushin (airgunster) wrote :

@chad.smith,
Hi Chad,

Q. I validated that the traceback is still seen in 18.2.4
A. Yep, correct, the issue is reproducible on cloud-init v. 18.2.4 as well (I just forgot to update this bug (apologize)).

Scott Moser (smoser) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=faa6f07e

Changed in cloud-init:
status: Confirmed → Fix Committed

This bug is believed to be fixed in cloud-init in version 18.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in cloud-init:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers