System locks up when using --freeze-filesystem /

[John Andrichak]

Using Ubuntu 11.04, when I execute ec2-consistent-snapshot --freeze-filesystem /, the command never returned, and the instance hung. This only occurred upon the first usage of ec2-consistent-snapshot on a freshly installed and configured AMI. After rebooting and running again, the command executed as expected.

[Eric Hammond (esh)]

I don't recommend freezing the root file system, especially if you have any sort of logging in place where the output from ec2-consistent snapshot might be sent to a file on the root file system as this could lock up the system.

Out of curiosity, what AMI id and instance type are you running?

[John Andrichak (jj4th)]

ami-06ad526f on a small instance. Unless one of the perl libraries used by ecs logs something, I'm pretty sure there's no logging going on. But it is possible that something related to the first run, some perl library trying to create a file, etc... (or something in ecs itself) might be causing the issue.

I definitely don't mind removing the --freeze-filesystem flag, but how do you take a consistent live root snapshot without locking the root file system?

[Eric Hammond (esh)]

I put data including databases on a separate EBS volume for consistent snapshots. I still snapshot the root file system just to get the resulting reduced failure rate for the EBS volume itself. I also make sure that I can rebuild the root file system from scratch on a new instance.

[Eric Hammond (esh)]

I am unable to reproduce using ami-06ad526f on m1.small

If you have a series of copy/paste commands that can be used to reproduce this, please let me know.

Make sure you do not post your secret credentials, but do include placeholders where they would go.

[Eric Hammond (esh)]

Thanks for taking the time to report this problem. Unfortunately, without further information I am unable to work on it. If you are able to provide instructions to reproduce the issue, please add them and re-open this bug report, or open a new one.

[John Andrichak (jj4th)]

I apologize, I meant to close this, but it almost certainly had to do with something else trying to access the root drive during the freeze. I do not believe it was at all related to ec2-consistent-snapshot other than that the freeze occurred at an inconvenient time.

[Eric Hammond (esh)]

John: Thanks for the note. I still think "--freeze-filesystem /" can be dangerous and may not be something that can be made safe in ec2-consistent-snapshot.

[Boyan Tabakov (bladealslayer)]

I can confirm this with ec2-consistent-snapshot v 0.44 on Ubuntu 12.04 LTS 64bit on Amazon EC2 with both micro and large instances. However, a simple restart didn't help in my case. What seemed to do the trick is run ec2-consistent-snapshot once with -n option (dry run). Indeed some file access (delayed library load?) seems to be the issue.

[Boyan Tabakov (bladealslayer)]

Correction. Dry run, doesn't actually help. The only way to make it work is manage to run the ec2-consistent-snapshot once without the --freeze-filesystem option.

Here's some debug information. The command freezes the filesystem, so I unfreeze it manually from a separate process. It shows some errors with failing Perl module compilation, probably due to frozen root filesystem:

ec2-consistent-snapshot: Using AWS access key: *************
ec2-consistent-snapshot: Using description 'footest'for all snapshot descriptions
ec2-consistent-snapshot: Thu Nov 22 10:35:48 2012: sync
ec2-consistent-snapshot: Thu Nov 22 10:35:48 2012: fsfreeze -f /
ec2-consistent-snapshot: Thu Nov 22 10:35:48 2012: create EC2 object
ec2-consistent-snapshot: Endpoint: https://ec2.eu-west-1.amazonaws.com
ec2-consistent-snapshot: volume_id: vol-******; description: footest-snap
ec2-consistent-snapshot: Thu Nov 22 10:35:48 2012: ec2-create-snapshot vol-****
ec2-consistent-snapshot: ERROR: create_snapshot: timeout
Compilation failed in require at /usr/share/perl5/URI/_server.pm line 2.
Compilation failed in require at /usr/share/perl5/URI/http.pm line 3.
Compilation failed in require at /usr/share/perl5/URI/https.pm line 2.
Compilation failed in require at (eval 539) line 2.
ec2-consistent-snapshot: Thu Nov 22 10:36:17 2012: fsfreeze -u /
fsfreeze: /: unfreeze failed: Invalid argument
ec2-consistent-snapshot: ERROR: fsfreeze -u /: failed(256)
ec2-consistent-snapshot: Thu Nov 22 10:36:17 2012: done