Unable to launch AWS M5 instance with newest Ubuntu AMI

Bug #1737187 reported by Rasmus Larsen on 2017-12-08

This bug report will be marked for expiration in 37 days if no further activity occurs. (find out why)

20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux-aws (Ubuntu)
Undecided
Unassigned

Bug Description

When booting Ubuntu Linux 16.04 on an M5.large instance, the instance fails to startup and dies, seemingly during initialization of the NVMe interface, here's the most relevant lines (full kernel log attached):

[ 8.840439] Btrfs loaded
[ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240939f1bb2, max_idle_ns: 440795263295 ns
[ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
[ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
[ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
[ 64.925737] nvme 0000:00:04.0: Removing after probe failure
[ 64.930506] iounmap: bad address ffffc90000e50000
[ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted 4.4.0-1041-aws #50-Ubuntu
[ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
[ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
[ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8 ffffffff813f7743
[ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8 ffffffff8106ae7f
[ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08 ffffffff8106aebc
[ 64.977878] Call Trace:
[ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
[ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
[ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
[ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30 [nvme]
[ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
[ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
[ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
[ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
[ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
[ 65.024806] [<ffffffff8143233a>] pci_stop_and_remove_bus_device_locked+0x1a/0x30
[ 65.032326] [<ffffffffc002909c>] nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
[ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
[ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
[ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
[ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
[ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
[ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
[ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
[ 65.071235] Trying to free nonexistent resource <00000000febf0000-00000000febf3fff>
[ 125.080477] random: nonblocking pool is initialized
(...End of output...)

Relevant data:
AMI id: ami-8fd760f6 (Should be the latest official release)
Instance type: m5.large
Region: eu-west-1

Issue seems completely repeatable.

Rasmus Larsen (rla-2) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-aws (Ubuntu):
status: New → Confirmed
Rene Cunningham (rene-q) wrote :

Is this still considered an issue?

I booted ami-0bc11fa490cf2ead7 (ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20190204.3) on an m5.large in eu-west-1 and it was fine.

Po-Hsu Lin (cypressyew) wrote :

Do you still experiencing this issue?

Changed in linux-aws (Ubuntu):
status: Confirmed → Incomplete
Download full text (3.5 KiB)

Hi,

I am sorry I have not tried recently, I moved my system to Centos to get
round the issue.

Regards
Mark

On Tue, 30 Jul 2019 at 09:31, Po-Hsu Lin <email address hidden> wrote:

> Do you still experiencing this issue?
>
> ** Changed in: linux-aws (Ubuntu)
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1737187
>
> Title:
> Unable to launch AWS M5 instance with newest Ubuntu AMI
>
> Status in linux-aws package in Ubuntu:
> Incomplete
>
> Bug description:
> When booting Ubuntu Linux 16.04 on an M5.large instance, the instance
> fails to startup and dies, seemingly during initialization of the NVMe
> interface, here's the most relevant lines (full kernel log attached):
>
> [ 8.840439] Btrfs loaded
> [ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
> 0x240939f1bb2, max_idle_ns: 440795263295 ns
> [ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
> [ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
> [ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
> [ 64.925737] nvme 0000:00:04.0: Removing after probe failure
> [ 64.930506] iounmap: bad address ffffc90000e50000
> [ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted
> 4.4.0-1041-aws #50-Ubuntu
> [ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
> [ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
> [ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8
> ffffffff813f7743
> [ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8
> ffffffff8106ae7f
> [ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08
> ffffffff8106aebc
> [ 64.977878] Call Trace:
> [ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
> [ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
> [ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
> [ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30
> [nvme]
> [ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
> [ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
> [ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
> [ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
> [ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
> [ 65.024806] [<ffffffff8143233a>]
> pci_stop_and_remove_bus_device_locked+0x1a/0x30
> [ 65.032326] [<ffffffffc002909c>]
> nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
> [ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
> [ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
> [ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
> [ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
> [ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
> [ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.071235] Trying to free nonexistent resource
> <0000000...

Read more...

Rasmus Larsen (rla-2) wrote :
Download full text (3.5 KiB)

No, I believe it's resolved.

On Tue, Jul 30, 2019, 10:31 Po-Hsu Lin <email address hidden> wrote:

> Do you still experiencing this issue?
>
> ** Changed in: linux-aws (Ubuntu)
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1737187
>
> Title:
> Unable to launch AWS M5 instance with newest Ubuntu AMI
>
> Status in linux-aws package in Ubuntu:
> Incomplete
>
> Bug description:
> When booting Ubuntu Linux 16.04 on an M5.large instance, the instance
> fails to startup and dies, seemingly during initialization of the NVMe
> interface, here's the most relevant lines (full kernel log attached):
>
> [ 8.840439] Btrfs loaded
> [ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
> 0x240939f1bb2, max_idle_ns: 440795263295 ns
> [ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
> [ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
> [ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
> [ 64.925737] nvme 0000:00:04.0: Removing after probe failure
> [ 64.930506] iounmap: bad address ffffc90000e50000
> [ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted
> 4.4.0-1041-aws #50-Ubuntu
> [ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
> [ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
> [ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8
> ffffffff813f7743
> [ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8
> ffffffff8106ae7f
> [ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08
> ffffffff8106aebc
> [ 64.977878] Call Trace:
> [ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
> [ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
> [ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
> [ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30
> [nvme]
> [ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
> [ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
> [ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
> [ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
> [ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
> [ 65.024806] [<ffffffff8143233a>]
> pci_stop_and_remove_bus_device_locked+0x1a/0x30
> [ 65.032326] [<ffffffffc002909c>]
> nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
> [ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
> [ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
> [ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
> [ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
> [ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
> [ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.071235] Trying to free nonexistent resource
> <00000000febf0000-00000000febf3fff>
> [ 125.080477] random: nonblocking pool is initial...

Read more...

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers