| 2012-11-14 21:54:58 |
Scott Moser |
bug |
|
|
added bug |
| 2012-11-14 21:54:58 |
Scott Moser |
attachment added |
|
console log of instance in un-reachable state https://bugs.launchpad.net/bugs/1078926/+attachment/3433598/+files/console-ami-be70f5d7-t1.micro.txt |
|
| 2012-11-14 21:57:13 |
Scott Moser |
attachment added |
|
console log of instance after stop-instances https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1078926/+attachment/3433600/+files/console-ami-be70f5d7-t1.micro-stopped.txt |
|
| 2012-11-14 21:57:59 |
Scott Moser |
attachment added |
|
cloud-init.log of instance https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1078926/+attachment/3433601/+files/cloud-init.log |
|
| 2012-11-14 21:59:36 |
Scott Moser |
affects |
cloud-init (Ubuntu) |
ubuntu |
|
| 2012-11-14 21:59:47 |
Scott Moser |
bug |
|
|
added subscriber Ben Howard |
| 2012-11-14 21:59:51 |
Scott Moser |
ubuntu: status |
New |
Confirmed |
|
| 2012-11-14 21:59:54 |
Scott Moser |
ubuntu: importance |
Undecided |
High |
|
| 2012-11-15 00:59:58 |
Shane Meyers |
bug |
|
|
added subscriber Shane Meyers |
| 2012-11-19 14:59:33 |
Scott Moser |
attachment added |
|
boot.log of failed instance. https://bugs.launchpad.net/ubuntu/+bug/1078926/+attachment/3438705/+files/boot.log |
|
| 2012-11-19 15:24:43 |
Scott Moser |
attachment added |
|
/var/log/syslog from instance https://bugs.launchpad.net/ubuntu/+bug/1078926/+attachment/3438716/+files/syslog |
|
| 2012-11-20 14:13:20 |
Stefan Bader |
attachment added |
|
Upstream patch under discussion https://bugs.launchpad.net/ubuntu/+bug/1078926/+attachment/3439485/+files/0001-xen-netfront-handle-compound-page-fragments-on-trans.patch |
|
| 2012-11-20 16:17:44 |
Ubuntu Foundations Team Bug Bot |
tags |
amd64 apport-bug ec2-images raring |
amd64 apport-bug ec2-images patch raring |
|
| 2012-11-21 22:00:32 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/raring-proposed/linux-ppc |
|
| 2012-11-21 23:06:33 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/raring-proposed/linux-lowlatency |
|
| 2012-11-23 15:27:10 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2012-11-27 17:45:27 |
Scott Moser |
attachment added |
|
good console log for reference https://bugs.launchpad.net/ubuntu/+bug/1078926/+attachment/3445276/+files/console-good.log |
|
| 2012-12-03 16:54:41 |
Scott Moser |
description |
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while cloud-init's log shows it WARNing, which should go to console.
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while
cloud-init's log shows it WARNing, which should go to console.
Related bugs:
* mountall: bug 1059471 2.41 fails to mount root partition
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
|
| 2012-12-03 18:04:12 |
Scott Moser |
description |
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while
cloud-init's log shows it WARNing, which should go to console.
Related bugs:
* mountall: bug 1059471 2.41 fails to mount root partition
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while
cloud-init's log shows it WARNing, which should go to console.
Related bugs:
* mountall: bug 1059471 2.41 fails to mount root partition
* plymouth: bug 1086072 some output to /dev/console does not reach /dev/console
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
|
| 2012-12-03 21:50:28 |
Steve Langasek |
affects |
ubuntu |
mountall (Ubuntu) |
|
| 2012-12-03 22:00:37 |
Steve Langasek |
mountall (Ubuntu): status |
Confirmed |
In Progress |
|
| 2012-12-03 22:00:37 |
Steve Langasek |
mountall (Ubuntu): assignee |
|
Steve Langasek (vorlon) |
|
| 2012-12-03 22:02:09 |
Steve Langasek |
nominated for series |
|
Ubuntu Quantal |
|
| 2012-12-03 22:02:09 |
Steve Langasek |
bug task added |
|
mountall (Ubuntu Quantal) |
|
| 2012-12-03 22:02:09 |
Steve Langasek |
nominated for series |
|
Ubuntu Raring |
|
| 2012-12-03 22:02:09 |
Steve Langasek |
bug task added |
|
mountall (Ubuntu Raring) |
|
| 2012-12-03 22:06:51 |
Steve Langasek |
mountall (Ubuntu Quantal): status |
New |
In Progress |
|
| 2012-12-03 22:06:53 |
Steve Langasek |
mountall (Ubuntu Quantal): importance |
Undecided |
High |
|
| 2012-12-03 22:06:55 |
Steve Langasek |
mountall (Ubuntu Quantal): assignee |
|
Steve Langasek (vorlon) |
|
| 2012-12-03 23:48:44 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/mountall |
|
| 2012-12-03 23:58:05 |
Steve Langasek |
description |
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while
cloud-init's log shows it WARNing, which should go to console.
Related bugs:
* mountall: bug 1059471 2.41 fails to mount root partition
* plymouth: bug 1086072 some output to /dev/console does not reach /dev/console
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
[Impact]
The previous SRU, while fixing the problem it was intended to fix, partially reintroduced the problem from before 2.41 where some filesystem events would end up blocking on one another when they shouldn't. This is particularly noticeable for filesystems that have been mounted by the initramfs and there are jobs started on the 'mounted' event for one of these. In the particular case of cloud-init, the jobs that start on mounted MOUNTPOINT=/ block waiting for the network to come up, which needs the 'virtual-filesystems' event which isn't happening because the 'mounted MOUNTPOINT=/run' event hasn't finished processing - it's behind the 'mounted /' in the event queue. This intermittently causes cloud boot times in excess of 5 minutes.
[Test case]
1. Boot the current quantal daily cloud images.
2. Confirm that at least sometimes, the images take 5 minutes to boot.
3. Upgrade mountall to the quantal-proposed version.
4. Confirm that the images now boot without hitting the "wait for network" timeout.
[Regression potential]
Minimal, as this is correcting a regression from the previous SRU.
This seems sporadic failure at best. I had seen it on openstack fail similarly.
Today I launched 7 instances of us-east-1 t1.micro from each of :
ami-be70f5d7 ebs/ubuntu-raring-daily-amd64-server-20121109
ami-de27a2b7 ebs/ubuntu-raring-daily-amd64-server-20121110
ami-f6c94d9f ebs/ubuntu-raring-daily-amd64-server-20121111
ami-cc8307a5 ebs/ubuntu-raring-daily-amd64-server-20121112
ami-5c43c735 ebs/ubuntu-raring-daily-amd64-server-20121113
and then another one of 20121109 and 20121113.
One of the 2 20121109 failed to boot, and I will attach its console log and cloud-init.log. The log was obtained from stopping the instance, attaching to another system, and getting it. Then I restarted the instance, and it booted fine.
Things that were of interest:
* from cloud-init's perspective, the thing that failed was finding the ec2 metadata service.
* In the failure case, there were no messages to the console log after initramfs, while
cloud-init's log shows it WARNing, which should go to console.
Related bugs:
* mountall: bug 1059471 2.41 fails to mount root partition
* plymouth: bug 1086072 some output to /dev/console does not reach /dev/console
ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: cloud-init 0.7.0-0ubuntu2
ProcVersionSignature: User Name 3.5.0-17.28-generic 3.5.5
Uname: Linux 3.5.0-17-generic x86_64
ApportVersion: 2.6.2-0ubuntu3
Architecture: amd64
Date: Wed Nov 14 21:30:05 2012
Ec2AMI: ami-be70f5d7
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1b
Ec2InstanceType: t1.micro
Ec2Kernel: aki-825ea7eb
Ec2Ramdisk: unavailable
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: cloud-init
UpgradeStatus: No upgrade log present (probably fresh install) |
|
| 2012-12-04 01:21:43 |
Scott Moser |
attachment added |
|
patch as per vorlon is 2.45->2.46 https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1078926/+attachment/3451094/+files/mountall-more-asyncer.patch |
|
| 2012-12-04 02:08:34 |
Adam Conrad |
mountall (Ubuntu Quantal): status |
In Progress |
Fix Committed |
|
| 2012-12-04 02:08:36 |
Adam Conrad |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2012-12-04 02:08:41 |
Adam Conrad |
bug |
|
|
added subscriber SRU Verification |
| 2012-12-04 02:08:43 |
Adam Conrad |
tags |
amd64 apport-bug ec2-images patch raring |
amd64 apport-bug ec2-images patch raring verification-needed |
|
| 2012-12-04 02:46:01 |
Adam Conrad |
nominated for series |
|
Ubuntu Precise |
|
| 2012-12-04 02:46:01 |
Adam Conrad |
bug task added |
|
mountall (Ubuntu Precise) |
|
| 2012-12-04 02:47:14 |
Adam Conrad |
mountall (Ubuntu Precise): status |
New |
Fix Committed |
|
| 2012-12-04 05:44:17 |
Launchpad Janitor |
mountall (Ubuntu Raring): status |
In Progress |
Fix Released |
|
| 2012-12-05 16:22:45 |
Steve Langasek |
mountall (Ubuntu Precise): status |
Fix Committed |
Fix Released |
|
| 2012-12-05 16:34:35 |
Steve Langasek |
tags |
amd64 apport-bug ec2-images patch raring verification-needed |
amd64 apport-bug ec2-images patch raring verification-done |
|
| 2012-12-12 04:42:31 |
Chris Halse Rogers |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2012-12-12 04:43:12 |
Launchpad Janitor |
mountall (Ubuntu Quantal): status |
Fix Committed |
Fix Released |
|
| 2013-12-11 02:04:17 |
Launchpad Janitor |
branch linked |
|
lp:debian/mountall |
|
| 2013-12-11 02:04:41 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-branches/ubuntu/precise/mountall/precise-proposed |
|
| 2013-12-11 02:05:06 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-branches/ubuntu/quantal/mountall/quantal-proposed |
|
