Unable to launch AWS M5 instance with newest Ubuntu AMI

[Rasmus Larsen]

When booting Ubuntu Linux 16.04 on an M5.large instance, the instance fails to startup and dies, seemingly during initialization of the NVMe interface, here's the most relevant lines (full kernel log attached):

[ 8.840439] Btrfs loaded
[ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240939f1bb2, max_idle_ns: 440795263295 ns
[ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
[ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
[ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
[ 64.925737] nvme 0000:00:04.0: Removing after probe failure
[ 64.930506] iounmap: bad address ffffc90000e50000
[ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted 4.4.0-1041-aws #50-Ubuntu
[ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
[ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
[ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8 ffffffff813f7743
[ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8 ffffffff8106ae7f
[ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08 ffffffff8106aebc
[ 64.977878] Call Trace:
[ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
[ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
[ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
[ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30 [nvme]
[ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
[ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
[ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
[ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
[ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
[ 65.024806] [<ffffffff8143233a>] pci_stop_and_remove_bus_device_locked+0x1a/0x30
[ 65.032326] [<ffffffffc002909c>] nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
[ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
[ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
[ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
[ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
[ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
[ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
[ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
[ 65.071235] Trying to free nonexistent resource <00000000febf0000-00000000febf3fff>
[ 125.080477] random: nonblocking pool is initialized
(...End of output...)

Relevant data:
AMI id: ami-8fd760f6 (Should be the latest official release)
Instance type: m5.large
Region: eu-west-1

Issue seems completely repeatable.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rene Cunningham (rene-q)]

Is this still considered an issue?

I booted ami-0bc11fa490cf2ead7 (ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20190204.3) on an m5.large in eu-west-1 and it was fine.

[Po-Hsu Lin (cypressyew)]

Do you still experiencing this issue?

[Mark Stratmann (stratmm)]

Hi,

I am sorry I have not tried recently, I moved my system to Centos to get
round the issue.

Regards
Mark

On Tue, 30 Jul 2019 at 09:31, Po-Hsu Lin <email address hidden> wrote:

> Do you still experiencing this issue?
>
> ** Changed in: linux-aws (Ubuntu)
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1737187
>
> Title:
> Unable to launch AWS M5 instance with newest Ubuntu AMI
>
> Status in linux-aws package in Ubuntu:
> Incomplete
>
> Bug description:
> When booting Ubuntu Linux 16.04 on an M5.large instance, the instance
> fails to startup and dies, seemingly during initialization of the NVMe
> interface, here's the most relevant lines (full kernel log attached):
>
> [ 8.840439] Btrfs loaded
> [ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
> 0x240939f1bb2, max_idle_ns: 440795263295 ns
> [ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
> [ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
> [ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
> [ 64.925737] nvme 0000:00:04.0: Removing after probe failure
> [ 64.930506] iounmap: bad address ffffc90000e50000
> [ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted
> 4.4.0-1041-aws #50-Ubuntu
> [ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
> [ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
> [ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8
> ffffffff813f7743
> [ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8
> ffffffff8106ae7f
> [ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08
> ffffffff8106aebc
> [ 64.977878] Call Trace:
> [ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
> [ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
> [ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
> [ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30
> [nvme]
> [ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
> [ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
> [ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
> [ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
> [ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
> [ 65.024806] [<ffffffff8143233a>]
> pci_stop_and_remove_bus_device_locked+0x1a/0x30
> [ 65.032326] [<ffffffffc002909c>]
> nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
> [ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
> [ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
> [ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
> [ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
> [ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
> [ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.071235] Trying to free nonexistent resource
> <0000000...

[Rasmus Larsen (rla-2)]

No, I believe it's resolved.

On Tue, Jul 30, 2019, 10:31 Po-Hsu Lin <email address hidden> wrote:

> Do you still experiencing this issue?
>
> ** Changed in: linux-aws (Ubuntu)
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1737187
>
> Title:
> Unable to launch AWS M5 instance with newest Ubuntu AMI
>
> Status in linux-aws package in Ubuntu:
> Incomplete
>
> Bug description:
> When booting Ubuntu Linux 16.04 on an M5.large instance, the instance
> fails to startup and dies, seemingly during initialization of the NVMe
> interface, here's the most relevant lines (full kernel log attached):
>
> [ 8.840439] Btrfs loaded
> [ 61.888098] clocksource: tsc: mask: 0xffffffffffffffff max_cycles:
> 0x240939f1bb2, max_idle_ns: 440795263295 ns
> [ 64.807909] nvme 0000:00:04.0: I/O 0 QID 0 timeout, disable controller
> [ 64.916220] nvme 0000:00:04.0: Cancelling I/O 0 QID 0
> [ 64.920779] nvme 0000:00:04.0: Identify Controller failed (-4)
> [ 64.925737] nvme 0000:00:04.0: Removing after probe failure
> [ 64.930506] iounmap: bad address ffffc90000e50000
> [ 64.934951] CPU: 1 PID: 32 Comm: kworker/1:1 Not tainted
> 4.4.0-1041-aws #50-Ubuntu
> [ 64.942528] Hardware name: Amazon EC2 m5.large/, BIOS 1.0 10/16/2017
> [ 64.947684] Workqueue: events nvme_remove_dead_ctrl_work [nvme]
> [ 64.952688] 0000000000000286 fd001be315a343d4 ffff8800bbb4bcd8
> ffffffff813f7743
> [ 64.960923] ffff880036c6d4f0 ffffc90000e50000 ffff8800bbb4bcf8
> ffffffff8106ae7f
> [ 64.969368] ffff880036c6d4f0 ffff880036c6d658 ffff8800bbb4bd08
> ffffffff8106aebc
> [ 64.977878] Call Trace:
> [ 64.981322] [<ffffffff813f7743>] dump_stack+0x63/0x90
> [ 64.985878] [<ffffffff8106ae7f>] iounmap.part.1+0x7f/0x90
> [ 64.990583] [<ffffffff8106aebc>] iounmap+0x2c/0x30
> [ 64.994873] [<ffffffffc002964a>] nvme_dev_unmap.isra.35+0x1a/0x30
> [nvme]
> [ 65.000071] [<ffffffffc002a73e>] nvme_remove+0xce/0xe0 [nvme]
> [ 65.005125] [<ffffffff81439309>] pci_device_remove+0x39/0xc0
> [ 65.009912] [<ffffffff815440e1>] __device_release_driver+0xa1/0x150
> [ 65.015009] [<ffffffff815441b3>] device_release_driver+0x23/0x30
> [ 65.020019] [<ffffffff814321ea>] pci_stop_bus_device+0x8a/0xa0
> [ 65.024806] [<ffffffff8143233a>]
> pci_stop_and_remove_bus_device_locked+0x1a/0x30
> [ 65.032326] [<ffffffffc002909c>]
> nvme_remove_dead_ctrl_work+0x3c/0x50 [nvme]
> [ 65.037731] [<ffffffff81099585>] process_one_work+0x165/0x480
> [ 65.042482] [<ffffffff810998eb>] worker_thread+0x4b/0x4c0
> [ 65.046978] [<ffffffff810998a0>] ? process_one_work+0x480/0x480
> [ 65.051905] [<ffffffff8109fa65>] kthread+0xe5/0x100
> [ 65.056224] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.061380] [<ffffffff8181728f>] ret_from_fork+0x3f/0x70
> [ 65.066103] [<ffffffff8109f980>] ? kthread_create_on_node+0x1e0/0x1e0
> [ 65.071235] Trying to free nonexistent resource
> <00000000febf0000-00000000febf3fff>
> [ 125.080477] random: nonblocking pool is initial...

[Launchpad Janitor (janitor)]

[Expired for linux-aws (Ubuntu) because there has been no activity for 60 days.]