Cannot encode a text script as base64 in write_files

[Adam Acosta]

Minimal user-data to reproduce:

#cloud-config
write_files:
- encoding: b64
  content: IyEvYmluL3NoCmVjaG8gaGVsbG8gd29ybGQ=
  path: /var/lib/cloud/scripts/per-once/hello.sh
  permissions: '0755'

This is the base64 encoding of:

#!/bin/sh
echo hello world

This will produce Errno 8 ENOEXEC. cloud-init will log 'Exec format error. Missing #! in script?' but that's because it's guessing. The actual reason is the shell will believe this to be a binary file (but not a recognized executable format). You can run file /var/lib/cloud/script/per-once/hello.sh and it will return 'Data' as the file type. If you use cat to read the file, it will work since cat auto-interprets as text, but if you use less, it will show the characters but with nonsense bytes between each.

The problem is the Python base64 module returns a bytes object, and if that is never decoded to UTF-8, writing it out in binary mode will write out a binary file. If it is decoded and written in text mode, it will work fine.

Suggested fix:

You can't blindly decode and interpret as text because that will break attempts to actually write out a binary file, but a key can be added to the write_files module indicating that a file is a text file. If that is present, decode the bytes object.

Workaround:

Don't base64 encode text files. This works fine using the yaml | string continuation feature instead.

Use case for base64 encoding:

I am creating a scriptable module to build VMs from installation media iso files utilizing cloud-init as the installer automation method, providing the ability to store scripts in a directory and generate a user-data file programmatically and then writing it to an iso for use as a NoCloud source. Imagine Packer but it works without needing to ssh to the VM to provision it, so you can build very minimal VMs that might not even have networking. It's marginally simpler to base64 encode all of the scripts provided in order to avoid having to deal with yaml indentation subtleties, but I can do this as-is with find and replace on all line starts in the scripts, figuring out how much indentation to add by tracking how much the "content" key had.

[Dan Watkins (oddbloke)]

Hi Adam,

Thanks for using cloud-init and for the bug report! I haven't been able to reproduce your issue locally:

$ cat user-data/lp1987872.yaml
#cloud-config
write_files:
- encoding: b64
  content: IyEvYmluL3NoCmVjaG8gaGVsbG8gd29ybGQ=
  path: /var/lib/cloud/scripts/per-once/hello.sh
  permissions: '0755'
$ lxc launch ubuntu:f -c user.user-data="$(cat user-data/lp1987872.yaml)"
Creating the instance
Instance name is: settled-octopus
Starting settled-octopus
$ lxc shell settled-octopus

... now within launched instance ...

# cat /var/lib/cloud/scripts/per-once/hello.sh
#!/bin/sh
echo hello world

# file $_
/var/lib/cloud/scripts/per-once/hello.sh: POSIX shell script, ASCII text executable

# /var/lib/cloud/scripts/per-once/hello.sh
hello world

# grep "Exec format error" /var/log/cloud-init.log || echo No lines found
No lines found

# grep "hello world" /var/log/cloud-init-output.log
hello world

It looks to me like this is working as expected. Could you run `cloud-init collect-logs` on an affected instance (and attach the tarball here), and give us more details about how we could reproduce this problem? Once done, please move this back to New.

Thanks!

Dan

[Launchpad Janitor (janitor)]

[Expired for cloud-init because there has been no activity for 60 days.]

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3875