Ubuntu 15.10 Amazon EC2 stock images seem to lead to corrupt custom images

[Alan]

I am used to launching a Canonical Ubuntu Amazon machine image, customizing with a build script, then creating my own AMI for quick launching. This process is leading to corrupt custom AMIs that don't boot correctly for 15.10 images. I wondered if this was bug 1510345 [1], but this is supposed to be fixed in the images I'm using. Steps to reproduce:
I use this image [2] as an example in us-east-1 Amazon EC2 region:
ami-1c552a76 099720109477/ubuntu/images/hvm/ubuntu-wily-15.10-amd64-server-20151106
however a prior Canonical 15.10 image yields the same result.
1. In us-east-1, run:
ec2run ami-1c552a76 -t m3.medium -f shutdown.sh --instance-initiated-shutdown-behavior stop
In this example, shutdown.sh just has two lines:
#! /bin/sh
shutdown -h now
An actual builder script installs packages, sets up users, etc. I've been using a process like this since Ubuntu 10.10.
2. Wait for the instance to stop.
3. Create the custom image.
ec2cim instance_id -n name -d description
4. Wait for the custom ami to become available.
5. Boot the new custom ami.
ec2run ami -t t2.micro
Console output is attached, but here are the last few lines:
[[32m OK [0m] Started Wait for all "auto" /etc/ne...be up for network-online.target.

[[32m OK [0m] Reached target Network is Online.

ifup-wait-all-auto.service

Welcome to emergency mode! After logging in, type "journalctl -xb" to view

system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to

try again to boot into default mode.

root@ip-172-31-55-220:~# [ 329.642713] reboot: Power down

6. Start the instance that was used to create the ami, it seems to boot correctly.

Troubleshooting steps:
1. I wanted to detach the root device and mount it on another VM to look at the startup logs, but I get a mount error.
2. I tried this, again, a tried and true recipe, with 15.04 images and it works. There is something different about the 15.10 images.

Notes:
[1] https://bugs.launchpad.net/ubuntu-on-ec2/+bug/1510345
[2] http://cloud-images.ubuntu.com/releases/wily/release/

[Ben Howard (darkmuggle-deactivatedaccount)]

Can you test the cloud-init I've uploaded to:
https://launchpad.net/~utlemming/+archive/ubuntu/lp1514485

I think that this is caused by a Bug #1514485

[Alan (8libra)]

Bug 1514234 is confirmed to be a manifestation of bug 1514485.

Test/workaround:
For cloud instances, before shutdown/migration/snapshotting, run:
sed -i "s/nobootwait/nofail/g" /etc/fstab
This will change the nobootwait option to nofail in /etc/fstab using
the sed stream editor. Snapshotted/migrated instances will boot with
this fix.

Thanks Ben Howard!!

On 11/10/15, Ben Howard <email address hidden> wrote:
> Can you test the cloud-init I've uploaded to:
> https://launchpad.net/~utlemming/+archive/ubuntu/lp1514485
>
> I think that this is caused by a Bug #1514485
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1514234
>
> Title:
> Ubuntu 15.10 Amazon EC2 stock images seem to lead to corrupt custom
> images
>
> Status in Ubuntu on EC2:
> New
>
> Bug description:
> I am used to launching a Canonical Ubuntu Amazon machine image,
> customizing with a build script, then creating my own AMI for quick
> launching. This process is leading to corrupt custom AMIs that don't boot
> correctly for 15.10 images. I wondered if this was bug 1510345 [1], but this
> is supposed to be fixed in the images I'm using. Steps to reproduce:
> I use this image [2] as an example in us-east-1 Amazon EC2 region:
>
> ami-1c552a76 099720109477/ubuntu/images/hvm/ubuntu-wily-15.10-amd64-server-20151106
> however a prior Canonical 15.10 image yields the same result.
> 1. In us-east-1, run:
> ec2run ami-1c552a76 -t m3.medium -f shutdown.sh
> --instance-initiated-shutdown-behavior stop
> In this example, shutdown.sh just has two lines:
> #! /bin/sh
> shutdown -h now
> An actual builder script installs packages, sets up users, etc. I've been
> using a process like this since Ubuntu 10.10.
> 2. Wait for the instance to stop.
> 3. Create the custom image.
> ec2cim instance_id -n name -d description
> 4. Wait for the custom ami to become available.
> 5. Boot the new custom ami.
> ec2run ami -t t2.micro
> Console output is attached, but here are the last few lines:
> [[32m OK [0m] Started Wait for all "auto" /etc/ne...be up for
> network-online.target.
>
>
> [[32m OK [0m] Reached target Network is Online.
>
>
> ifup-wait-all-auto.service
>
>
> Welcome to emergency mode! After logging in, type "journalctl -xb" to
> view
>
> system logs, "systemctl reboot" to reboot, "systemctl default" or ^D
> to
>
> try again to boot into default mode.
>
> root@ip-172-31-55-220:~# [ 329.642713] reboot: Power down
>
>
> 6. Start the instance that was used to create the ami, it seems to boot
> correctly.
>
> Troubleshooting steps:
> 1. I wanted to detach the root device and mount it on another VM to look
> at the startup logs, but I get a mount error.
> 2. I tried this, again, a tried and true recipe, with 15.04 images and it
> works. There is something different about the 15.10 images.
>
> Notes:
> [1] https://bugs.launchpad.net/ubuntu-on-ec2/+bug/1510345
> [2] http://cloud-images.ubuntu.com/releases/wily/release/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu-on-ec2/+bug/1514234/+subscriptions
>

[Alan (8libra)]

The 20151116.1 series of Wily EC2 images uploaded by Canonical has
resolved this bug. I think this bug can be closed. Example of a
resolved image:
ami-dfcab0b5 099720109477/ubuntu/images/hvm/ubuntu-wily-15.10-amd64-server-20151116.1

On 11/10/15, Alan Tu <email address hidden> wrote:
> Bug 1514234 is confirmed to be a manifestation of bug 1514485.
>
> Test/workaround:
> For cloud instances, before shutdown/migration/snapshotting, run:
> sed -i "s/nobootwait/nofail/g" /etc/fstab
> This will change the nobootwait option to nofail in /etc/fstab using
> the sed stream editor. Snapshotted/migrated instances will boot with
> this fix.
>
> Thanks Ben Howard!!
>
>
> On 11/10/15, Ben Howard <email address hidden> wrote:
>> Can you test the cloud-init I've uploaded to:
>> https://launchpad.net/~utlemming/+archive/ubuntu/lp1514485
>>
>> I think that this is caused by a Bug #1514485
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1514234
>>
>> Title:
>> Ubuntu 15.10 Amazon EC2 stock images seem to lead to corrupt custom
>> images
>>
>> Status in Ubuntu on EC2:
>> New
>>
>> Bug description:
>> I am used to launching a Canonical Ubuntu Amazon machine image,
>> customizing with a build script, then creating my own AMI for quick
>> launching. This process is leading to corrupt custom AMIs that don't boot
>> correctly for 15.10 images. I wondered if this was bug 1510345 [1], but
>> this
>> is supposed to be fixed in the images I'm using. Steps to reproduce:
>> I use this image [2] as an example in us-east-1 Amazon EC2 region:
>>
>> ami-1c552a76 099720109477/ubuntu/images/hvm/ubuntu-wily-15.10-amd64-server-20151106
>> however a prior Canonical 15.10 image yields the same result.
>> 1. In us-east-1, run:
>> ec2run ami-1c552a76 -t m3.medium -f shutdown.sh
>> --instance-initiated-shutdown-behavior stop
>> In this example, shutdown.sh just has two lines:
>> #! /bin/sh
>> shutdown -h now
>> An actual builder script installs packages, sets up users, etc. I've
>> been
>> using a process like this since Ubuntu 10.10.
>> 2. Wait for the instance to stop.
>> 3. Create the custom image.
>> ec2cim instance_id -n name -d description
>> 4. Wait for the custom ami to become available.
>> 5. Boot the new custom ami.
>> ec2run ami -t t2.micro
>> Console output is attached, but here are the last few lines:
>> [[32m OK [0m] Started Wait for all "auto" /etc/ne...be up for
>> network-online.target.
>>
>>
>> [[32m OK [0m] Reached target Network is Online.
>>
>>
>> ifup-wait-all-auto.service
>>
>>
>> Welcome to emergency mode! After logging in, type "journalctl -xb" to
>> view
>>
>> system logs, "systemctl reboot" to reboot, "systemctl default" or ^D
>> to
>>
>> try again to boot into default mode.
>>
>> root@ip-172-31-55-220:~# [ 329.642713] reboot: Power down
>>
>>
>> 6. Start the instance that was used to create the ami, it seems to
>> boot
>> correctly.
>>
>> Troubleshooting steps:
>> 1. I wanted to detach the root device and mount it on another VM to
>> look
>> at the startup logs, but I get a mount error.
>> 2. I tried ...