vagrant artful64 box filesystem too small

[Berislav Lopac]

[SRU Justfication, Artful]

[Impact]

Booting the 4.13 Artful kernel with vagrant using VirtualBox trips the warning:

[ 61.010337] VFS: brelse: Trying to free free buffer
[ 61.114875] ------------[ cut here ]------------
[ 61.114886] WARNING: CPU: 0 PID: 683 at /build/linux-XO_uEE/linux-4.13.0/fs/buffer.c:1205 __brelse+0x21/0x30

and a failed resize of a partition. The root cause has been bisected down to the following commmit:

commit c20cfc27a47307e811346f85959cf3cc07ae42f9
Author: Christoph Hellwig <email address hidden>
Date: Wed Apr 5 19:21:07 2017 +0200

    block: stop using blkdev_issue_write_same for zeroing

[Fix]
The Upstream commit directly fixes this issue:

commit d5ce4c31d6df518dd8f63bbae20d7423c5018a6c
Author: Ilya Dryomov <email address hidden>
Date: Mon Oct 16 15:59:10 2017 +0200

    block: cope with WRITE ZEROES failing in blkdev_issue_zeroout()

..however we also require a backport of the following upstream commit to apply the above commit cleanly.

commit 425a4dba7953e35ffd096771973add6d2f40d2ed
Author: Ilya Dryomov <email address hidden>
Date: Mon Oct 16 15:59:09 2017 +0200

    block: factor out __blkdev_issue_zero_pages()

[Testscase]

On Ubuntu Xenial:

1. sudo apt-get install virtualbox vagrant
2. edit /etc/group and add one's user name to the vboxusers group
3. log out log back
4. vagrant init ubuntu/artful64
5. vagrant up
6. vagrant ssh
7. dmesg | grep "VFS: brelse"

without the fix one will see the VFS brelse warning message and the / partition will not have been resized.

with a fixed system there is is no VFS vbrelse warning and / as been resized as expected.

[Regresion potential]
These patches touch the blk library so potentially it could break the block layer and corrupt data on disk. However these are upstream fixes that address the buggy commit c20cfc27a47307e811346f85959cf3cc07ae42f9 and are known to address the bug.

-------------------------------

After building a new vagrant instance using the ubuntu/artful64 box (v20171023.1.0), the size of the filesystem seems to be much too small. Here's the output of `df -h` on the newly built instance:

vagrant@ubuntu-artful:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 991M 0 991M 0% /dev
tmpfs 200M 3.2M 197M 2% /run
/dev/sda1 2.2G 2.1G 85M 97% /
tmpfs 999M 0 999M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 999M 0 999M 0% /sys/fs/cgroup
vagrant 210G 182G 28G 87% /vagrant
tmpfs 200M 0 200M 0% /run/user/1000

For comparison, here is the same from the latest zesty64 box:

ubuntu@ubuntu-zesty:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 992M 0 992M 0% /dev
tmpfs 200M 3.2M 197M 2% /run
/dev/sda1 9.7G 2.5G 7.3G 26% /
tmpfs 999M 0 999M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 999M 0 999M 0% /sys/fs/cgroup
vagrant 210G 183G 28G 88% /vagrant
tmpfs 200M 0 200M 0% /run/user/1000

With artful64, the size of /dev/sda1 is reported as 2.2G, which results in 97% of disk usage immediately after building, even though the disk size is 10G, as reported by the fdisk:

vagrant@ubuntu-artful:~$ sudo fdisk -l
Disk /dev/sda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4ad77c39

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 20971486 20969439 10G 83 Linux

Disk /dev/sdb: 10 MiB, 10485760 bytes, 20480 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Almost any additional installation results in a "No space left on device" error.

[Dan Watkins (oddbloke)]

Hi Berislav,

Thanks for your bug report! All the Vagrant boxes ship like that, and
the filesystem is resized on first boot by cloud-init[0]. Could you
attach /var/lib/cloud-init.log from a Vagrant instance?

Thanks!

Dan

[0] This allows for the image to be used with multiple different disk
    sizes more easily.

[Berislav Lopac (b11c)]

I'm getting this:

vagrant@ubuntu-artful:~$ cat /var/lib/cloud-init.log
cat: /var/lib/cloud-init.log: No such file or directory

[Berislav Lopac (b11c)]

Ah, I found /var/log/cloud-init.log :)

[Berislav Lopac (b11c)]

Here is the cloud-init.log

[Scott Moser (smoser)]

The attached /var/log/cloud-init.log has 2 boots in it.
One starts at 14:15:18 (line 1) and one starting at 14:21:17,808 (line 719).

The first boot successfully updated the partition table for /dev/sda
so that the first partition (/dev/sda1) took the whole ~ 10G disk.

  2017-10-24 14:15:33,338 - cc_growpart.py[INFO]: '/' resized: changed (/dev/sda, 1) from 2359296000 to 10736352768
then cloud-init runs 'resize2fs /dev/sda1' and gets which exited with
exit code 1 and prints the following to stderr.
 resize2fs: Remote I/O error While checking for on-line resizing support

The second boot finds that there is no need for growpart to run (NOCHANGE)
But it stil calls 'resize2fs /dev/sda1', and that fails the same way.

So, good news is this looks like it is recreatable.
I suspect you can see the same error just by running:
  sudo resize2fs /dev/sda1

Bad news is that I'm not sure what could be going wrong.

Could you collect a 'dmesg' output?

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1726818

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Berislav Lopac (b11c)]

This is the dmesg output.

[Scott Moser (smoser)]

So the/some interesting points of dmesg.log:

[ 0.000000] random: get_random_bytes called from start_kernel+0x42/0x4e1 with crng_init=0
[ 0.000000] Linux version 4.13.0-16-generic (buildd@lcy01-02) (gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu2)) #19-Ubuntu SMP Wed Oct 11 18:35:14 UTC 2017 (Ubuntu 4.13.0-16.19-generic 4.13.4)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.13.0-16-generic root=LABEL=cloudimg-rootfs ro console=tty1 console=ttyS0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] XSAVE consistency problem, dumping leaves
[ 0.000000] ------------[ cut here ]------------
[ 0.000000] WARNING: CPU: 0 PID: 0 at /build/linux-XO_uEE/linux-4.13.0/arch/x86/kernel/fpu/xstate.c:596 fpu__init_system_xstate+0x437/0x7f0
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 4.13.0-16-generic #19-Ubuntu
[ 0.000000] task: ffffffff8ec10480 task.stack: ffffffff8ec00000
[ 0.000000] RIP: 0010:fpu__init_system_xstate+0x437/0x7f0
[ 0.000000] RSP: 0000:ffffffff8ec03d88 EFLAGS: 00010082 ORIG_RAX: 0000000000000000
[ 0.000000] RAX: 0000000000000029 RBX: 000000000000000a RCX: ffffffff8ec5fcc8
[ 0.000000] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000047
[ 0.000000] RBP: ffffffff8ec03de8 R08: 6f63204556415358 R09: 0000000000000007
[ 0.000000] R10: ffffffff8ee5b7c0 R11: 0000000000000000 R12: 0000000000000340
[ 0.000000] R13: 0000000000000440 R14: ffffffff8ec03da8 R15: 0000000000000100
[ 0.000000] FS: 0000000000000000(0000) GS:ffffffff8ee7f000(0000) knlGS:0000000000000000
[ 0.000000] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.000000] CR2: ffff88000008a000 CR3: 0000000006113000 CR4: 00000000000406a0
[ 0.000000] Call Trace:
[ 0.000000] ? get_cpu_cap+0x2a3/0x370
[ 0.000000] ? msr_clear_bit+0x3a/0xa0
[ 0.000000] fpu__init_system+0x1fb/0x248
[ 0.000000] early_cpu_init+0x145/0x14c
[ 0.000000] ? 0xffffffff8de00000
[ 0.000000] setup_arch+0xbf/0xcd4
[ 0.000000] ? printk+0x52/0x6e
[ 0.000000] ? early_idt_handler_array+0x120/0x120
[ 0.000000] start_kernel+0xab/0x4e1
[ 0.000000] ? early_idt_handler_array+0x120/0x120
[ 0.000000] x86_64_start_reservations+0x24/0x26
[ 0.000000] x86_64_start_kernel+0x13e/0x161
[ 0.000000] secondary_startup_64+0x9f/0x9f
[ 0.000000] Code: 0f 85 42 fd ff ff 44 3b 25 44 e9 22 00 74 2c 80 3d 32 8e f6 ff 00 75 15 48 c7 c7 68 f3 a8 8e c6 05 22 8e f6 ff 01 e8 cf 6f 03 ff <0f> ff 83 3d f5 be d7 ff 00 74 05 e8 eb b4 f8 fe 48 8b 35 47 ed
[ 0.000000] ---[ end trace 97d576cdb74ee0b3 ]---
[ 0.000000] CPUID[0d, 00]: eax=00000007 ebx=00000440 ecx=00000440 edx=00000000
...
[ 23.952218] ioc0: LSI53C1030 A0: Capabilities={Initiator}
[ 24.340974] scsi host2: ioc0: LSI53C1030 A0, FwRev=00000000h, Ports=1, MaxQ=256, IRQ=20
[ 24.720543] scsi 2:0:0:0: Direct-Access VBOX HARDDISK 1.0 PQ: 0 ANSI: 5
[ 25.360635] scsi target2:0:0: Beginning Domain Validation
[ 25.760433] floppy0: no floppy controllers found
[ 25.760450] floppy0: floppy_shutdown: timeout handl...

[Scott Moser (smoser)]

ApportVersion: 2.20.7-0ubuntu3
Architecture: amd64
DistroRelease: Ubuntu 17.10
Ec2AMI: ami-000002da
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.large
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
JournalErrors:
 Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system.
       Users in the 'systemd-journal' group can see all messages. Pass -q to
       turn off this notice.
 No journal files were opened due to insufficient permissions.
Package: linux
PackageArchitecture: amd64
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=C.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Tags: artful ec2-images
Uname: Linux 4.13.0-16-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

[Scott Moser (smoser)]

apport information

[Scott Moser (smoser)]

apport information

[Ryan Harper (raharper)]

This thread shows up with that xsave warn:

https://www.virtualbox.org/pipermail/vbox-dev/2017-May/014466.html

Which suggests adding 'noxsave' to the kernel command line as a workaround.

[Yasuo Honda (yahonda)]

Hi, I am getting the same problem.

> Which suggests adding 'noxsave' to the kernel command line as a workaround.

I have been looking for how to set 'noxsave' kernel boot parameter using Vagrant but
unable to find it. Does anyone know how to configure it?

Thank you.

[Scott Moser (smoser)]

Yasuo,
It is not exposed in any way through vagrant.
It is a kernel command line option.
You'd have to edit /etc/default/grub and add it to GRUB_CMDLINE_LINUX_DEFAULT and then run 'update-grub'.

I had done a quick test and didn't seem to think that it fixed the issue, but I'd be interested in hearing your feedback.

[Yasuo Honda (yahonda)]

Thanks for the feedback. I am also unable to address the root file system size
by updating /etc/default/grub as below.

### At Vagrant host

$ vagrant init ubuntu/artful64
$ vagrant up
$ vagrant ssh

### At Vagrant guest

1. Root file syste size is 2.2G

vagrant@ubuntu-artful:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.2G 895M 1.3G 42% /

2. Edit /etc/default/grub and modify GRUB_CMDLINE_LINUX_DEFAULT parameter

vagrant@ubuntu-artful:~$ sudo vi /etc/default/grub

- Before change
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0"

- After change
GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0 noxsave"

3. Execute update-grub

vagrant@ubuntu-artful:~$ sudo update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.13.0-16-generic
Found initrd image: /boot/initrd.img-4.13.0-16-generic
done
vagrant@ubuntu-artful:~$

4. Reboot the Vagrant guest

vagrant@ubuntu-artful:~$ sudo reboot

5. Root file system size does not change.

$ vagrant ssh

vagrant@ubuntu-artful:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.2G 896M 1.3G 42% /
vagrant@ubuntu-artful:~$

Thanks.

[Colin Ian King (colin-king)]

From the fs/buffer.c:1205 __brelse+0x21/0x30 it is impossible to deduce what is broken. I'm trying to reproduce this using vagrant and VirtualBox. Can you inform me of the vagrant box you are using and the full set of instructions on how one can reproduce this. Some pertinent facts to help me reproduce this are:

1. What host system are you using?
2. What vagrant box are you using? (any URLs or copies of the .box file or would be handy)
3. Exactly what is the vagrant setup being used.

I'm not totally familiar with using vagrant, so clear instructions so I can set up a reproducer to help me debug this further would be very helpful. Thanks.

[Christian Ehrhardt  (paelzer)]

Hi cking,
I rarely use it, but for me (on Xenial) the following worked.

$ sudo apt install vagrant virtualbox
$ vagrant init ubuntu/artful64
# note that you have to elminate all KVMs before you can do the next step
$ vagrant up
$ vagrant ssh

vagrant@ubuntu-artful:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.2G 895M 1.3G 42% /

[Christian Ehrhardt  (paelzer)]

With the above in comment #17 I see the limited size after boot and the crash in dmesg.
But then a repro with
  $ sudo resize2fs /dev/sda1
doesn't work - well it works, but that is the problem it works to resize without triggering the issue "again".

[Berislav Lopac (b11c)]

1. Verified on both Ubuntu Linux 17.10 and Windows 10
2. The official ubuntu/artful64 box: https://app.vagrantup.com/ubuntu/boxes/artful64. The problem is verified with the currently latest version (v20171115.1.0)
3. Verified with the minimal vagrant configuration (see attached Vagrantfile).

Steps to reproduce:

0. install Vagrant and VirtualBox
1. $ mkdir artful-test
2. (copy the attached Vagrantfile to artful-test)
3. $ cd artful-test
4. $ vagrant up
5. $ vagrant ssh
6. (on vagrant-box)$ df -h

Result will be something like:

Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 99M 3.1M 96M 4% /run
/dev/sda1 2.2G 896M 1.3G 42% /
tmpfs 495M 0 495M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 495M 0 495M 0% /sys/fs/cgroup
vagrant 210G 73G 138G 35% /vagrant
tmpfs 99M 0 99M 0% /run/user/1000

Expected: /dev/sda1 to have a size of around 10GB
Actual: /dev/sda1 has a size of 2.2GB

[Berislav Lopac (b11c)]

I can confirm @paelzer's comment above, resize2fs after vagrant up works. In fact this helps me in the short term, as adding `resize2fs /dev/sda1` to the beginning of my provision script resolves my issue.

But this is just a workaround and not a real solution.

[Colin Ian King (colin-king)]

Thanks, can reproduce the issue now, will debug

[Colin Ian King (colin-king)]

git bisect'd to:

c20cfc27a47307e811346f85959cf3cc07ae42f9 is the first bad commit
commit c20cfc27a47307e811346f85959cf3cc07ae42f9
Author: Christoph Hellwig <email address hidden>
Date: Wed Apr 5 19:21:07 2017 +0200

    block: stop using blkdev_issue_write_same for zeroing

    We'll always use the WRITE ZEROES code for zeroing now.

    Signed-off-by: Christoph Hellwig <email address hidden>
    Reviewed-by: Martin K. Petersen <email address hidden>
    Reviewed-by: Hannes Reinecke <email address hidden>
    Signed-off-by: Jens Axboe <email address hidden>

[Colin Ian King (colin-king)]

Whilst one cannot revert this for 4.13, I did remove the offending code, rebuilt and now one can boot without the kernel failure and the hdd is resized correctly.

[Colin Ian King (colin-king)]

I'll try and figure out *why* this patch is causing the kernel grief tomorrow.

[Colin Ian King (colin-king)]

My current hunch is that this is due to blk optimization functionality that landed in the kernel and perhaps the virt blk device on VirtualBox does not support it yet.

[jxn (jacksonscottsly)]

I believe this also affects the bionic64 nightlies.

[Stefan Bader (smb)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Yasuo Honda (yahonda)]

I have validated that / file system size becomes 9.7GB
after using `-proposed` option as suggested.

# Steps to validate

- At Vagrant host
$ vagrant init ubuntu/artful64
$ vagrant ssh

- At Vagrant guest without `-proposed`

vagrant@ubuntu-artful:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 2.2G 1.3G 862M 60% /

-
$ sudo vi /etc/apt/sources.list and add the following entry.
```
deb http://archive.ubuntu.com/ubuntu/ artful-proposed restricted main multiverse universe
```

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo reboot

$ vagrant ssh
vagrant@ubuntu-artful:~$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.7G 1.4G 8.4G 14% /

Thank you for the fix.

[Yasuo Honda (yahonda)]

Removed `verification-needed-artful` and added `verification-done-artful` tag.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.13.0-38.43

---------------
linux (4.13.0-38.43) artful; urgency=medium

  * linux: 4.13.0-38.43 -proposed tracker (LP: #1755762)

  * Servers going OOM after updating kernel from 4.10 to 4.13 (LP: #1748408)
    - i40e: Fix memory leak related filter programming status
    - i40e: Add programming descriptors to cleaned_count

  * [SRU] Lenovo E41 Mic mute hotkey is not responding (LP: #1753347)
    - platform/x86: ideapad-laptop: Increase timeout to wait for EC answer

  * fails to dump with latest kpti fixes (LP: #1750021)
    - kdump: write correct address of mem_section into vmcoreinfo

  * headset mic can't be detected on two Dell machines (LP: #1748807)
    - ALSA: hda/realtek - Support headset mode for ALC215/ALC285/ALC289
    - ALSA: hda - Fix headset mic detection problem for two Dell machines
    - ALSA: hda - Fix a wrong FIXUP for alc289 on Dell machines

  * CIFS SMB2/SMB3 does not work for domain based DFS (LP: #1747572)
    - CIFS: make IPC a regular tcon
    - CIFS: use tcon_ipc instead of use_ipc parameter of SMB2_ioctl
    - CIFS: dump IPC tcon in debug proc file

  * i2c-thunderx: erroneous error message "unhandled state: 0" (LP: #1754076)
    - i2c: octeon: Prevent error message on bus error

  * hisi_sas: Add disk LED support (LP: #1752695)
    - scsi: hisi_sas: directly attached disk LED feature for v2 hw

  * EDAC, sb_edac: Backport 1 patch to Ubuntu 17.10 (Fix missing DIMM sysfs
    entries with KNL SNC2/SNC4 mode) (LP: #1743856)
    - EDAC, sb_edac: Fix missing DIMM sysfs entries with KNL SNC2/SNC4 mode

  * [regression] Colour banding and artefacts appear system-wide on an Asus
    Zenbook UX303LA with Intel HD 4400 graphics (LP: #1749420)
    - drm/edid: Add 6 bpc quirk for CPT panel in Asus UX303LA

  * DVB Card with SAA7146 chipset not working (LP: #1742316)
    - vmalloc: fix __GFP_HIGHMEM usage for vmalloc_32 on 32b systems

  * [Asus UX360UA] battery status in unity-panel is not changing when battery is
    being charged (LP: #1661876) // AC adapter status not detected on Asus
    ZenBook UX410UAK (LP: #1745032)
    - ACPI / battery: Add quirk for Asus UX360UA and UX410UAK

  * ASUS UX305LA - Battery state not detected correctly (LP: #1482390)
    - ACPI / battery: Add quirk for Asus GL502VSK and UX305LA

  * support thunderx2 vendor pmu events (LP: #1747523)
    - perf pmu: Extract function to get JSON alias map
    - perf pmu: Pass pmu as a parameter to get_cpuid_str()
    - perf tools arm64: Add support for get_cpuid_str function.
    - perf pmu: Add helper function is_pmu_core to detect PMU CORE devices
    - perf vendor events arm64: Add ThunderX2 implementation defined pmu core
      events
    - perf pmu: Add check for valid cpuid in perf_pmu__find_map()

  * lpfc.ko module doesn't work (LP: #1746970)
    - scsi: lpfc: Fix loop mode target discovery

  * Ubuntu 17.10 crashes on vmalloc.c (LP: #1739498)
    - powerpc/mm/book3s64: Make KERN_IO_START a variable
    - powerpc/mm/slb: Move comment next to the code it's referring to
    - powerpc/mm/hash64: Make vmalloc 56T on hash

  * ethtool -p fails to light NIC LED on HiSilicon D05 systems (LP: #1748567)
    - net...