cloud-init corrupting grub

[Jared Baker]

I believe I have discovered problems in the *recent builds* of Ubuntu 14.04 and 16.04 (NO ISSUE with 18.04 or OLDER 14.04/16.04 builds) cloud images (https://cloud-images.ubuntu.com/) for instances with very large local disks. We are using OpenStack Queens on Ubuntu 16.04 and have noticed that when our users deploy our largest instance type with ~5TB local disk that the VM deploys but when that VM is rebooted for the first time it wont boot and becomes stuck at the 'grub rescue' prompt with 'error: unknown filesystem.'.

I believe the issue is happening in the growpart function of cloud-init of the first boot of the VM to grow the root partition to fit the large local disk. I have attached as much logging and output information from 14.04, 16.04 and 18.04 but I do not have a developer background so I can't pinpoint the issue entirely.

I am able to easily reproduce this issue and would happy to help with the resolution of this bug in any capacity.

Thank you!

[Scott Moser (smoser)]

seems likely related to bug 1762748, and maybe that change overlapping with bug 1666853 ?

[Jared Baker (shubjero)]

Hi Scott,

Yes probably related bugs for sure. Again, I'd be happy to help test a fix if one is developed as I'm able to easily reproduce this problem.

[Scott Moser (smoser)]

There are 2 issues at play here:
a. bug 1666853: grub2 does not work correctly in 16.04 with ext4 filesystems larger than 2TB. This appears to be fixed in 18.04, but the bug is not currently marked as fixed there.

b. bug 1762748 was SRU'd to 16.04 which exposed bug 1666853. Before that fix, the filesystem would not get correctly grown and the bug would not be seen.

So there are two paths forward here:
1.) revert the change in bug 1762748 in 16.04 so that we do not expose bug 1666853.
2.) locate and SRU a fix for bug 1666853 to 16.04 so that large disks function as desired.