Unexpected partition growth on first boot on impish for raspberry pi

[Noah]

Hi,

On Raspberry Pi since Impish, the partition always grows even if I set the following in user-data of cloud-init.

growpart:
  mode: off
  devices: ['/']

I have tested this on 21.04, and it works, but is broken on 21.10. (partition always grows)

I've also tested this in KVM on amd64, and it works (partition does NOT grow).

This is a problem for me because I am using runcmd in cloud init to migrate my drive to LVM/LUKS, and the partitioning step fails because the drive is already full.

Cheers,
Noah

[Chad Smith (chad.smith)]

Thank you for filing a bug and making cloud-init and Ubuntu better. Can you run `sudo cloud-init collect-logs` and attach that tarfile to this bug for ease of triage? Note that cloud-init collect-logs potentially grabs user-data. If you have sensitive credentials defined in your user-data, please check and redact as necessary.

[Noah (noahod)]

Hi, attached are the requested logfiles.

Cheers,
Noah

[James Falcon (falcojr)]

According to the logs, growpart isn't doing any resizing:

2021-10-13 13:33:05,609 - stages.py[DEBUG]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) with frequency always
2021-10-13 13:33:05,610 - handlers.py[DEBUG]: start: init-network/config-growpart: running config-growpart with frequency always
2021-10-13 13:33:05,610 - helpers.py[DEBUG]: Running config-growpart using lock (<cloudinit.helpers.DummyLock object at 0xffff7fe84be0>)
2021-10-13 13:33:05,610 - cc_growpart.py[DEBUG]: growpart disabled: mode=False
2021-10-13 13:33:05,610 - handlers.py[DEBUG]: finish: init-network/config-growpart: SUCCESS: config-growpart ran successfully

Does adding this to your cloud-config work?

resize_rootfs: false

In cloud-init, the growpart module is followed by the resizefs module used to grow the root filesystem.
https://cloudinit.readthedocs.io/en/latest/topics/modules.html#resizefs

If that doesn't fix your problem, is there something else on your system that could be modifying your partition size?

[Noah (noahod)]

My system is a fresh image downloaded from here: https://cdimage.ubuntu.com/releases/21.10/release/ubuntu-21.10-preinstalled-server-arm64+raspi.img.xz

I flashed it with Balana Etcher on OS X and overwrote user-data on the system-boot partition with the attached file,

With your suggestion of resize_rootfs: false, the partition still grows but the filesystem doesn't, which doesn't solve the issue because I want the partition not to grow.

I'll attach photos of the boot which shows growroot and the partition sizes in a followup comment.

The same user-data file with ubuntu-21.04-preinstalled-server-arm64+raspi.img.xz flashed *doesn't* have any issues, growroot does not happen with 21.04, only 21.10.

[Noah (noahod)]

Attached photo showing boot process

[Noah (noahod)]

Attached photo showing partition / filesystem sizing

[Noah (noahod)]

Also worth noting that this didn't happen when I tested on KVM, the bug is somehow related to the raspberry pi image

[James Falcon (falcojr)]

There's a separate package called cloud-initramfs-growroot (separate from cloud-init), that has a tool called growroot. It can be disabled by touching /etc/growroot-disabled . If you have a "x-systemd.growfs" in your /etc/fstab, that's probably what's initiating it.

[Noah (noahod)]

ok.. but I can't touch that from MacOS which doesn't have ext4 filesystem support.. Or is there some way to touch that with cloud-init before it happens?

Why did the behavior change between releases of Ubuntu and how can I hope to get this issue resolved? Should I create a new bug for a different package?

[James Falcon (falcojr)]

I'm shooting from the hip here, but you could try adding this to your cloud-config:

bootcmd:
 - touch /etc/growroot-disabled

If that doesn't work, you could try a bug targeting the https://bugs.launchpad.net/ubuntu/+source/linux-raspi project.

[Noah (noahod)]

Thanks, that's a good suggestion, but it didn't work.

Can I just reassign this ticket to the linux-raspi project?

[Juerg Haefliger (juergh)]

linux-raspi is the kernel package but this isn't a kernel problem

[Noah (noahod)]

I tried this again on the latest Raspberry Pi daily for Jammy, downloaded from here today (20210216):
https://cdimage.ubuntu.com/ubuntu-server/daily-preinstalled/current/jammy-preinstalled-server-arm64+raspi.img.xz

The issue persists. Can anyone suggest how I can get this assigned to the right project? I'm trying ubuntu-image as a target, as I think that's how the raspberry pi images are generated? Sorry if it's the wrong package.

[Juerg Haefliger (juergh)]

The reason for this behavior is that cloud-initramfs-growroot is installed since Impish which adds an initramfs hook to do the resizing, long before cloud-init runs. Disabling it in user-data won't do anything.

I thought cloud-initramfs-growroot was solely used with old kernels that can't resize mounted filesystems but I could be wrong. Wondering though why we install that package all of a sudden.

[Scott Moser (smoser)]

@Noah,

The image you pointed at there has the package 'cloud-initramfs-growroot'
in its initramfs. cloud-initramfs-growroot is going to run growpart on the
root filesystem during the initramfs unless one of the following files
exists on the root filesystem:

   /var/lib/cloud/instance/root-grown
   /etc/growroot-disabled
   /etc/growroot-grown

So the easiest to do is to create one of those files. You have mentioned
above that MacOS does not have ext4 filesystem support. So you can't easily
do it natively.

Here are some other options:

1. use libguestfs/guestfish or some other mechanism to boot a linux in which
you could create one of those files.

2. Improve growpart to read a kernel command line option to disable itself.

3. Improve growpart to further look for a "disable" file on a vfat
filesystem labelled 'system-boot' (lets just say 'growroot-disabled'), and
then use MacOS support of vfat to create the file there.

4. Further partition the image before booting. Growpart can only grow the
root filesystem until the next partition boundary.

Further exploring option '4', here is some information about the image:

a.) there are 2 partitions,
  * 1 : 256M partition with VFAT filesystem labelled 'system-boot'.
  * 2 : ~3.9GB partition with ext4 filesystem labelled 'writable'
b.) there is 16676864 bytes of unpartitioned space after partition 2.
c.) These are just rants:
 * The second partition starts on a MiB boundary, but does not end on one.
   So when you create another partition on that disk you're likely
 * The partition table type really should be GPT, not dos. 10 years ago
   GPT would have been the right decision and it still is today.
 * I wonder why 16676864 bytes (32572 sectors) of unused space. that is an
   strange number.

So something you probably can do is to add a partition of minimal size
directly after the last partition. You could use fdisk to do this
interactively or sfdisk like below:

If I were trying to do this right, I'd end my mini-partition on a MiB
boundary so that the fourth partition would start on a MiB boundary.
MiB boundary just make sense for alignment (or even 4MiB boundary, for
LVM extent alignment. Maybe it isn't completely necessary, but why not?).

But for simplicitly sake, lets just add a partition right at the end of the
previous one.

Info is attached and at https://paste.ubuntu.com/p/SmQKM2qCXX/ .

Below is the fdisk session inline here (launchpad doesn't format things
like that well, so I put it in attachemnt and pastebin also).

The end result is we have a partition (number 4) that starts right where
the second partition ends, so that:
 a.) you can use partition 3 as you wanted before.
 b.) growpart can't grow anything.

-----
$ fdisk jammy-preinstalled-server-arm64+raspi.img

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk jammy-preinstalled-server-arm64+raspi.img: 4.18 GiB, 4483710976 bytes, 8757248 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type...

[Scott Moser (smoser)]

> I thought cloud-initramfs-growroot was solely used with old kernels that can't resize mounted
> filesystems but I could be wrong. Wondering though why we install that package all of a sudden.

that is another option, to remove cloud-initramfs-growroot from the
image/initrd. The problem with that is that you're then relying on
cloud-init to grow the rootfs, and if you disable cloud-init you don't get
the grow.

I'm not exactly sure what the purpose of these images is.

[Noah (noahod)]

Thanks for looking into this!

I was hoping to create an IoT package that was agnostic of which device I deploy it on to, and cloud-init is great for this because it gives me a universal API for provisioning that's supposed to work across all ubuntu flavors and even other distros...

So for example, what might one day deployed to a Raspberry Pi, might be tomorrow deployed to an Intel board, or a RiskV board... My user-data script would pull in Docker and Docker would take care of the platform agnostic fetching of images.. (eg, ARM or Intel)

With this in mind it would be great to fix cloud-init...

I've been going round in circles trying to find the package list where cloud-initramfs-growroot has been added to... I tried looking in https://github.com/snapcore/pi-gadget but I think that's only for ubuntu-core and not the preinstalled server images..

Does anyone have ideas?

[Scott Moser (smoser)]

@Noah,
cloud-init isn't doing anything wrong. Its working as designed.

'growroot', which is provided by cloud-initramfs-tools (upstream [1], package [2]) also didn't do anything wrong. It's sole purpose in the initramfs is to do what it is doing.

I'm not sure what code creates the image you've pointed to. That is the code that included growroot into the image. Thats what Juerg was saying.

It should not be necessary with any kernel newer than 3.8 (~10 years ago). If growroot was *not* present, then the user-data you have provided would tell cloud-init not to grow the partition.

--
[1] https://code.launchpad.net/cloud-initramfs-tools
[2] https://launchpad.net/ubuntu/+source/cloud-initramfs-tools

[Noah (noahod)]

Thanks, I understand that cloud-init package isn't doing anything wrong.

I've cross-posted here in the hopes that someone can give me a lead as to where to find what changed between releases and how to proceed.

https://github.com/snapcore/pi-gadget/issues/85

Cheers

[Juerg Haefliger (juergh)]

@noahod, I'm looking into it.

[Dave Jones (waveform)]

Hmmm, I wonder if this is in the seeds, given we *finally* got around to using the platform seeds to build the impish image (the intention was to do this in hirsute, and they were ready but for some reason they were never activated on that release).

Ah ha! Here's the culprit:

https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/platform/tree/raspi-common

Now I wonder why that was added ...

https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/platform/commit/raspi-common?id=66ab1060453f5cdc8db8af901910282de20411bc

Ohhh. I wonder if this is required to grow the partition on the desktop images. Those don't use cloud-init at all, instead there's a "x-systemd.growfs" tag in the fstab, but perhaps this package was required to operate that? I thought that was systemd-growfs though... Either way, it seems like something that could be added to the desktop seed specifically instead of the raspi-common seed.

[Dimitri John Ledkov (xnox)]

systemd stuff did either partition or fs but not both.

we used the cloud initramfs implementation on the desktop, because yes, it doesn't do cloud-init.

probably moving that out of the common seed will help.

[Dave Jones (waveform)]

@xnox Ah, that makes sense, thanks. I'll propose some seed updates in a bit.

[Noah (noahod)]

I'm worried about this being baked into the 22.04 LTS and then I can't use a stable LTS Raspberry Pi distro I can use for another 3 years or so..

Is there anything I can do to help?

[Dave Jones (waveform)]

Sorry, this slipped off my radar -- I'll get on with the required PRs today

[Noah (noahod)]

Thank you so much! :)

[Łukasz Zemczak (sil2100)]

Not an expert in seeds, but the changes made sense. Merged and pushed!

[Noah (noahod)]

Should these changes be in the daily's already?

I just checked with https://cdimage.ubuntu.com/ubuntu-server/daily-preinstalled/current/jammy-preinstalled-server-arm64+raspi.img.xz downloaded today,

And got the same result as before, growroot runs and expands the partition..

[Juerg Haefliger (juergh)]

Did you disable it in user-data?

cloud-initramfs-growroot is no longer installed per the manifest https://cdimage.ubuntu.com/ubuntu-server/daily-preinstalled/current/jammy-preinstalled-server-amd64.manifest

[Juerg Haefliger (juergh)]

Ah never mind I looked at the amd64 image manifest, not raspi :-(

[Noah (noahod)]

I tried
growpart:
  mode: off
  devices: ['/']

and separately:
resize_rootfs: false

[Dave Jones (waveform)]

@noahod I've just tried this with the current https://cdimage.ubuntu.com/ubuntu-server/daily-preinstalled/current/jammy-preinstalled-server-arm64+raspi.img.xz image (downloaded today) and with just your growpart config (I didn't try the resize_rootfs key), the card booted without resizing the rootfs:

  ubuntu@ubuntu:~$ df -h /
  Filesystem Size Used Avail Use% Mounted on
  /dev/mmcblk0p2 3.3G 2.2G 964M 70% /

  ubuntu@ubuntu:~$ sudo fdisk -l /dev/mmcblk0
  Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: dos
  Disk identifier: 0xc34fdafd

  Device Boot Start End Sectors Size Id Type
  /dev/mmcblk0p1 * 2048 526335 524288 256M c W95 FAT32 (LBA)
  /dev/mmcblk0p2 526336 7656787 7130452 3.4G 83 Linux

Could you have a go with just the growpart config?

[Noah (noahod)]

It worked! I must have flashed the wrong image, Thank you so much for taking the time! And Thanks to everyone that worked on this!