The latest version creates /\"/var/lib/hibinit-agent\" directory

[Alex L. Demidov]

[Impact]
A state-dir directory is created to store instance state for AWS EC2 instances. This package parses a configuration file variable with quotes and makes the quotes part of the full path, thus creating a directory titles '"' in root.

[Fix]
Remove the quotes in the configuration file. See upstream commit https://github.com/aws/amazon-ec2-hibinit-agent/commit/d87d6365009f1663247fd8595b487c59d9b121f1

[Test Case]
# General test case
1.
# Create AWS EC2 instance
AWS_REGION="eu-north-1"
AWS_AMI="ami-07ca74b1a97901682"
AWS_KEY="your key name"
AWS_SECURITY_GROUP="security instance that allows ssh from anywhere"
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=focal-test-hibinit}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

2. SSH into instance

3. Remove weird directory if it exists
$ sudo rm -rf /'"'/

4. Start hibinit-agent
$ sudo systemctl start hibinit-agent

5. check for dir
# before fix
$ ll / | head -n 3
total 4096080
drwxr-xr-x 3 root root 4096 Jun 20 22:25 "/
drwxr-xr-x 20 root root 4096 Jun 20 22:25 ./
# after fix
$ ll / | head -n 3
total 4096076
drwxr-xr-x 19 root root 4096 Jun 20 22:25 ./
drwxr-xr-x 19 root root 4096 Jun 20 22:25 ../

# On upgrade
This test case ensures that /"/ is uninstalled on upgrade during from d/postinst.
1. Create an AWS instance
AWS_REGION="eu-north-1"
AWS_AMI="ami-07ca74b1a97901682"
AWS_KEY="your key name"
AWS_SECURITY_GROUP="security instance that allows ssh from anywhere"
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=focal-test-hibinit}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

2. Check for directory (ec2-hibinit-agent will be installed by default, and therefore directory will exist by default)
$ file /\"/
/"/: directory

3. Upgrade ec2-hibinit-agent
$ # add ppa if needed
$ sudo apt update -y && sudo apt upgrade ec2-hibinit-agent -y
--- snippet from upgrade ---
The following packages will be upgraded:
  ec2-hibinit-agent
1 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Need to get 12.6 kB of archives.
After this operation, 2048 B of additional disk space will be used.
Get:1 https://ppa.launchpadcontent.net/mitchdz/ec2-hibinit-agent-locale-and-quotes-fix/ubuntu lunar/main amd64 ec2-hibinit-agent all 1.0.0-0ubuntu13.2~lunar6 [12.6 kB]
Fetched 12.6 kB in 0s (82.2 kB/s)
(Reading database ... 125081 files and directories currently installed.)
Preparing to unpack .../ec2-hibinit-agent_1.0.0-0ubuntu13.2~lunar6_all.deb ...
Unpacking ec2-hibinit-agent (1.0.0-0ubuntu13.2~lunar6) over (1.0.0-0ubuntu13.1) ...
Setting up ec2-hibinit-agent (1.0.0-0ubuntu13.2~lunar6) ...
Installing new version of config file /etc/hibinit-config.cfg ...
Removing file /"/var/lib/hibinit-agent"/hibernation-enabled
Recursively removing directories /"/var/lib/hibinit-agent"
---

4. check for erroneous directory existence
$ file /\"/
/"/: cannot open `/"/' (No such file or directory)

[Where things can go wrong]
Past logic could look for the weird directory path (did test for this, and nothing along these lines were seen), on upgrade old path remains in filesystem without being deleted.

[Original Report]
The latest version 1.0.0-0ubuntu9.3 creates incorrect state-dir directory which includes double quotes in its name - /\"/var/lib/hibinit-agent\"/. I can see this behavior on Focal. See also upstream commit
https://github.com/aws/amazon-ec2-hibinit-agent/commit/d87d6365009f1663247fd8595b487c59d9b121f1

[Mitchell Dzurick (mitchdz)]

Thank you for reporting this Alex! I tested version 1.0.0-0ubuntu9.3 in a Focal EC2 instance (instructions on what I used to provision system below) and do not see this. Do you happen to have the AMI ID/machine configuration that you used and reproduce this? I'd be glad to try that configuration out.

What I used to test:
```
AWS_REGION="eu-north-1"
AWS_AMI="ami-07ca74b1a97901682"
AWS_KEY="your key name"
AWS_SECURITY_GROUP="security instance that allows ssh from anywhere"
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=focal-test-hibinit}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'
```

[Mitchell Dzurick (mitchdz)]

nevermind, I was able to reproduce this on my end!

[Mitchell Dzurick (mitchdz)]

This is a bug that affects all supported distributions and thus will be fixed for all of them. Example below is how to reproduce on Focal:

1.
# Create AWS EC2 instance
AWS_REGION="eu-north-1"
AWS_AMI="ami-07ca74b1a97901682"
AWS_KEY="your key name"
AWS_SECURITY_GROUP="security instance that allows ssh from anywhere"
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=focal-test-hibinit}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

2. SSH into instance

3. Remove weird directory if it exists
$ sudo rm -rf /'"'/

4. Start hibinit-agent
$ sudo systemctl start hibinit-agent

5. check for dir
# before fix
$ ll / | head -n 3
total 4096080
drwxr-xr-x 3 root root 4096 Jun 20 22:25 "/
drwxr-xr-x 20 root root 4096 Jun 20 22:25 ./
# after fix
$ ll / | head -n 3
total 4096076
drwxr-xr-x 19 root root 4096 Jun 20 22:25 ./
drwxr-xr-x 19 root root 4096 Jun 20 22:25 ../

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu15

---------------
ec2-hibinit-agent (1.0.0-0ubuntu15) mantic; urgency=medium

    * debian/patches/lp2023924-remove-quotes-from-state-dir.patch: fixes
      an issue where a weird directory would be created in root
      (LP: #2023924)
    * debian/patches/lp2024505-fix-locale-issue.patch: fixes an issue where
      certain locales can cause an error when trying to decode the output of
      `df -P` (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mitchell Dzurick (mitchdz)]

I've proposed the change for Focal/Jammy/Kinetic/Lunar (see attached MPs). Waiting to get approved into proposed now.

[Steve Langasek (vorlon)]

I don't see any code here that cleans up this wrong directory on upgrade for users affected by the bug?

[Mitchell Dzurick (mitchdz)]

Good catch Steve, I'll add that in d/preinst to the MP soon. Will update when that's done.

[Mitchell Dzurick (mitchdz)]

Steve,

I added the logic to remove /\"/ in d/preinst in the lunar MP - https://code.launchpad.net/~mitchdz/ubuntu/+source/ec2-hibinit-agent/+git/ec2-hibinit-agent/+merge/445090 since the code is almost the same for the other releases, let me know how this looks for lunar and then I'll add it to the rest of the releases once it looks good.

[Tired Sysadmin (wearyofallthiscrap)]

Mitchell,

1) The patch in that link looks like it's modifying postinst, not preinst. Dunno whether that makes a difference for purposes of the discussion; for that matter, I might also be misunderstanding how the patches are applied.

More importantly,

2) In that patch, the removal is only triggered if the hibernation-enabled file is present. On the systems where I've encountered the bug, the *file* never gets created because we're not using hibernation, but the misplaced *directory* is still created. Shouldn't the condition instead be to test for the existence of the *directory*? The current body of the 'if' would still be the correct action.

looking forward to this hitting the 22.04 builds!

[Mitchell Dzurick (mitchdz)]

Sorry about that, had a discussion with Steve offline and decided to keep the logic in d/postinst, just because it makes things more simple in certain cases.

Thanks for mentioning how your system looks! I'll add the logic to handle the scenario where the directory exists but not the file.

[Mitchell Dzurick (mitchdz)]

Logic updated.

[Steve Langasek (vorlon)]

Thanks, this looks good, but we should have a test case here that confirms the postinst change does what is intended.

[Mitchell Dzurick (mitchdz)]

Thanks Steve. Added a section "# On upgrade" in the [Test Case] section of the SRU template. Let me know how that looks.

[Steve Langasek (vorlon)]

Hello Alex, or anyone else affected,

Accepted ec2-hibinit-agent into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu13.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Alex, or anyone else affected,

Accepted ec2-hibinit-agent into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu12.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Alex, or anyone else affected,

Accepted ec2-hibinit-agent into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu11.22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Alex, or anyone else affected,

Accepted ec2-hibinit-agent into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-hibinit-agent/1.0.0-0ubuntu9.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Mitchell Dzurick (mitchdz)]

I went ahead and tested the proposed package for Lunar/Kinetic/Jammy/Focal. The results are good. The testing steps are very similar for all releases so I'll make a single comment and highlight the small changes between them.

I tested the following packages:
Lunar - 1.0.0-0ubuntu13.2
Kinetic - 1.0.0-0ubuntu12.2
Jammy - 1.0.0-0ubuntu11.22.04.3
Focal - 1.0.0-0ubuntu9.4

1) Create AWS instance like so:
aws ec2 run-instances \
 --region ${AWS_REGION} \
 --image-id ${AWS_AMI} \
 --count 1 \
 --instance-type t3.micro \
 --key-name ${AWS_KEY} \
 --metadata-options "HttpTokens=required" \
 --security-group-ids ${AWS_SECURITY_GROUP} \
 --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=test-hibinit-${RELEASE}}]' \
 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"Encrypted":true,"DeleteOnTermination":true,"VolumeSize":8,"VolumeType":"gp2"}}]' \
 --hibernation-options 'Configured=true'

Following AMIs were used:
images=(
  "ami-095495524a76a487a lunar"
  "ami-070f3a579ee524cc4 kinetic"
  "ami-086178f8d4f1d44a2 jammy"
  "ami-00ab811808de193f8 focal"
)

2) ssh to system & update
$ sudo apt update -y && sudo apt upgrade -y
...
$ file /\"/var/lib/hibinit-agent\"/
/"/var/lib/hibinit-agent"/: directory

3) update to proposed package
$ echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe" | sudo tee /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
$ sudo apt update -y && sudo apt upgrade -y

4) test proposed package
# For Lunar I had to specify the version like so:
# sudo apt install ec2-hibinit-agent=1.0.0-0ubuntu13.2
$ file /\"/var/lib/hibinit-agent\"/
/"/var/lib/hibinit-agent"/: cannot open `/"/var/lib/hibinit-agent"/' (No such file or directory)

In addition to this, I tested the hibernation functionality of the utility to ensure no regressions.
1) Create a program to run continuously and run it
/bin/cat <<EOM >~/allocate_mem.py
#!/usr/bin/python3
import time

# Allocate 200MB chunk of memory
size = 200 * 1024 * 1024 # 200MB
memory_chunk = bytearray(size)

print("Allocated 200MB of memory.")

# Enter indefinite loop
while True:
    time.sleep(1) # Wait for 1 second

# The script will never reach this point
EOM
python3 ~/allocate_mem.py &
2) hibernate instance, and then resume instance
3) check for process still running
$ ps aux | grep allocate_mem | grep -v grep
ubuntu 2151 0.1 22.1 220644 210992 ? S 20:08 0:00 python3 /home/ubuntu/allocate_mem.py

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu9.4

---------------
ec2-hibinit-agent (1.0.0-0ubuntu9.4) focal; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ec2-hibinit-agent has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu12.2

---------------
ec2-hibinit-agent (1.0.0-0ubuntu12.2) kinetic; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu11.22.04.3

---------------
ec2-hibinit-agent (1.0.0-0ubuntu11.22.04.3) jammy; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-hibinit-agent - 1.0.0-0ubuntu13.2

---------------
ec2-hibinit-agent (1.0.0-0ubuntu13.2) lunar; urgency=medium

    * d/p/lp2023924-remove-quotes-from-state-dir.patch: fixes an issue where a
      weird directory would be created in root (LP: #2023924)
    * d/preinst: remove the directory /" if it exists
    * d/p/lp2024505-fix-locale-issue.patch: fixes an issue where certain locales
      can cause an error when trying to decode the output of `df -P`
      (LP: #2024505)

 -- Mitchell Dzurick <email address hidden> Tue, 20 Jun 2023 17:01:17 -0700