Change encoding to use the system locale instead of ascii

[Mitchell Dzurick]

[Impact]
Some Locale will cause hibinit-agent to error out because the script tries to decode using ASCII. An example would be the russian locale ru_RU.UTF-8

[Fix]
One line fix from upstream - https://github.com/aws/amazon-ec2-hibinit-agent/commit/2392e860c7a75c2b61baef749bb90e79a2a3e81e

[Test Case]
1. Create an EC2 instance
AWS_REGION="eu-north-1"
AWS_AMI="ami-07ca74b1a97901682"
AWS_KEY="your key name"
AWS_SECURITY_GROUP="security instance that allows ssh from anywhere"
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=focal-test-hibinit}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

2. SSH into instance
3. install russian language and set locale
$ sudo apt install -y language-pack-ru
$ export LC_ALL=ru_RU.UTF-8
4. Attempt to use hibinit-agent
$ sudo /usr/bin/hibinit-agent
Traceback (most recent call last):
  File "/usr/bin/hibinit-agent", line 269, in swap_needs_touch
    dev = find_device_for_file(swap_place)
  File "/usr/bin/hibinit-agent", line 148, in find_device_for_file
    df_out = check_output(['df', '-P', filename]).decode('ascii')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/hibinit-agent", line 480, in <module>
    main()
  File "/usr/bin/hibinit-agent", line 413, in main
    config = Config(config_file, args)
  File "/usr/bin/hibinit-agent", line 309, in __init__
    swap_needs_touch(SWAP_FILE))
  File "/usr/bin/hibinit-agent", line 273, in swap_needs_touch
    raise Exception("Failed to find the filesystem type of /")
Exception: Failed to find the filesystem type of /

When fixed, the results should look like:
$ sudo /usr/bin/hibinit-agent
Effective config: {'log_to_syslog': True, 'log_to_stderr': True, 'mkswap': 'mkswap {swapfile}', 'swapon': 'swapon {swapfile}', 'swapoff': 'swapoff {swapfile}', 'touch_swap': False, 'grub_update': True, 'swap_percentage': 100, 'swap_mb': 4000, 'state_dir': '/var/lib/hibinit-agent'}
Requesting new IMDSv2 token.
Hibernation Configured Flag found
Will check if swap is at least: 4000 megabytes
There's sufficient swap available (have 4194304000, need 4194304000)
Running: swapon /swap-hibinit
Updating the kernel offset for the swapfile: /swap-hibinit
Setting swap device to 66305 with offset 1017856
Done updating the swap offset. Turning swapoff
Running: swapoff /swap-hibinit

[ Where things can go wrong ]
This now relies on the Python sys.getfilesystemencoding() to provide the correct value. However this is a standard library function so it is most likely very robust.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu15

---------------
ec2-hibinit-agent (1.0.0-0ubuntu15) mantic; urgency=medium

    * debian/patches/lp2023924-remove-quotes-from-state-dir.patch: fixes
      an issue where a weird directory would be created in root
      (LP: #2023924)
    * debian/patches/lp2024505-fix-locale-issue.patch: fixes an issue where
      certain locales can cause an error when trying to decode the output of
      `df -P` (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Steve Langasek (vorlon)]

FWIW it was not at all obvious to me from the python documentation thet getfilesystemencoding() would return the encoding for the *locale*. But:

$ sudo apt install -y language-pack-ja
$ LC_ALL=ja_JP python3 -c 'import sys; print(sys.getfilesystemencoding())'
euc_jp
$

So this DTRT.

[Steve Langasek (vorlon)]

Hello Mitchell, or anyone else affected,

Accepted ec2-hibinit-agent into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu13.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Mitchell, or anyone else affected,

Accepted ec2-hibinit-agent into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu12.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Mitchell, or anyone else affected,

Accepted ec2-hibinit-agent into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu11.22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Mitchell, or anyone else affected,

Accepted ec2-hibinit-agent into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu9.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mitchell Dzurick (mitchdz)]

I went ahead and tested the proposed package for Lunar/Kinetic/Jammy/Focal. The results are good. The testing steps are very similar for all releases so I'll make a single comment and highlight the small changes between them.

I tested the following packages:
Lunar - 1.0.0-0ubuntu13.2
Kinetic - 1.0.0-0ubuntu12.2
Jammy - 1.0.0-0ubuntu11.22.04.3
Focal - 1.0.0-0ubuntu9.4

1) Create AWS instance like so:
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=test-hibinit-${RELEASE}}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

Following AMIs were used:
images=(
  "ami-095495524a76a487a lunar"
  "ami-070f3a579ee524cc4 kinetic"
  "ami-086178f8d4f1d44a2 jammy"
  "ami-00ab811808de193f8 focal"
)

2) update system
$ sudo apt install -y language-pack-ru

3) set russian locale
$ export LC_ALL=ru_RU.UTF-8

4) test failure
$ sudo /usr/bin/hibinit-agent
Traceback (most recent call last):
  File "/usr/bin/hibinit-agent", line 269, in swap_needs_touch
    dev = find_device_for_file(swap_place)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/hibinit-agent", line 148, in find_device_for_file
    df_out = check_output(['df', '-P', filename]).decode('ascii')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/hibinit-agent", line 480, in <module>
    main()
  File "/usr/bin/hibinit-agent", line 413, in main
    config = Config(config_file, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/hibinit-agent", line 309, in __init__
    swap_needs_touch(SWAP_FILE))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/hibinit-agent", line 273, in swap_needs_touch
    raise Exception("Failed to find the filesystem type of /")
Exception: Failed to find the filesystem type of /

5) now try proposed package
$ echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe" | sudo tee /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
$ sudo apt update -y && sudo apt upgrade -y
$ sudo /usr/bin/hibinit-agent
Effective config: {'log_to_syslog': True, 'log_to_stderr': True, 'mkswap': 'mkswap {swapfile}', 'swapon': 'swapon {swapfile}', 'swapoff': 'swapoff {swapfile}', 'touch_swap': False, 'grub_update': True, 'swap_percentage': 100, 'swap_mb': 4000, 'state_dir': '/var/lib/hibinit-agent'}
Found '/var/lib/hibinit-agent/hibernation-enabled', configuring hibernation
Will check if swap is at least: 4000 megabytes
There's sufficient swap available (have 4194304000, need 4194304000)
Running: swapon /swap-hibinit
Updating the kernel offset for the swapfile: /swap-hibinit
Setting swap device to 66305 wi...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu9.4

---------------
ec2-hibinit-agent (1.0.0-0ubuntu9.4) focal; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ec2-hibinit-agent has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu12.2

---------------
ec2-hibinit-agent (1.0.0-0ubuntu12.2) kinetic; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu11.22.04.3

---------------
ec2-hibinit-agent (1.0.0-0ubuntu11.22.04.3) jammy; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu13.2

---------------
ec2-hibinit-agent (1.0.0-0ubuntu13.2) lunar; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/preinst: remove the directory /" if it exists
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700