vvfat core dump when enabling RW

[zappacor]

[Impact]

 * enabling vvfat with rw set breaks qemu once it writes something

 * Fix is a backport of a small upstream fix

[Test Case]

 * 1. get a guest that works (via uvtoools-libvirt or whatever you prever)
 * 2. get the qemu commandline that it is started with, in my case:
     $ sudo kvm -m 1024 -drive file=/dev/sdb,format=raw,cache=none,index=0,media=disk
 * 3. create a dir and add a file to check to share in vvfat mode later
   $ mkdir /tmp/sharevvfat
   $ echo host > /tmp/sharevvfat/hostfoo
 * 4. shutdown guest and start it again with the path as vvfat set to share rw mode, to do so append
     -drive format=vvfat,file=fat:rw:/tmp/sharevvfat/,if=virtio
 * in the guest mount the block device and write to it
     $ mount /dev/vdc1 /mnt
     $ echo guest > /mnt/guestfoo
     $ sync

 Currently crashes:
qemu-system-x86_64: /build/qemu-iZpOAh/qemu-2.5+dfsg/block/vvfat.c:2290: commit_direntries: Assertion `!strncmp(s->directory.pointer, "QEMU", 4)' failed. Aborted (core dumped)

 Expected: work to write files from guest and getting no fails/crashes

[Regression Potential]

 * I'd consider the potential next-to-none for general qemu/kvm use cases
   and low for cases using vvfat for the following reasons:
   * The fix is local and only executed in vvfat code, so affecting other
     qemu use cases should be next to impossible
   * code is upstream and in further Ubuntu Releases quite a while without
      issues showing up
   * General regression checks showed no issue

[Other Info]

 * n/a

---

Hi guys,

I'm getting this qemu crash message:
>>> qemu-system-x86_64: /build/qemu-TziMIO/qemu-2.5+dfsg/block/vvfat.c:2290: commit_direntries: Assertion `!strncmp(s->directory.pointer, "QEMU", 4)' failed.
>>> Aborted (core dumped)
when launching qemu with this options for a VVFAT drive:
>>> -drive file=fat:rw:./ROOT,if=virtio
(same happens when using cache=none and/or if=ide)

"uname -a" system info is:
>>> Linux RJZ-WRK-LNX 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
and "qemu --version" is:
>>> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.10), Copyright (c) 2003-2008 Fabrice Bellard

Not sure what logs to attach but I'll be glad to upload whatever needed.

Thanks in advance for you help,
Rolando

[Thomas Huth (th-huth)]

Moving to QEMU-Ubuntu since you're not using upstream QEMU (and the bug should have been fixed there as pointed out by Hervé on the qemu-devel mailing list).

[Christian Ehrhardt  (paelzer)]

The mentioned commit is:

commit ebb72c9f066e5f85259e1541a6d3fb5bfd6e73ff
Author: Kevin Wolf <email address hidden>
Date: Wed Apr 27 14:11:38 2016 +0200

    vvfat: Fix volume name assertion

    Commit d5941dd made the volume name configurable, but it didn't consider
    that the rw code compares the volume name string to assert that the
    first directory entry is the volume name. This made vvfat crash in rw
    mode.

    This fixes the assertion to compare with the configured volume name
    instead of a literal string.

    Cc: <email address hidden>
    Signed-off-by: Kevin Wolf <email address hidden>
    Reviewed-by: Markus Armbruster <email address hidden>
    Reviewed-by: Stefan Hajnoczi <email address hidden>

The fix is present since v2.6.0
As you're using v2.5.0, can you try with a newer QEMU version?

[Christian Ehrhardt  (paelzer)]

That means it is fixed in Yakkety and later, and not Broken in Trusty yet as the feature is not in.

[Christian Ehrhardt  (paelzer)]

Thank you Rolando for your report!
I think the issue is valid and a backport as SRU [1] possible.

I don't think you need to upload further Data, yet there is more you can help with.
For the SRU and testing it we will need some simplified steps to reproduce (and verify the fix).
Also see the Section SRU Template in [1].

So if you could add your steps to create the vfat image that then lets you trigger this that might help - I doubt it in this particular case, but sometimes suboptions on those steps are important so it is always better to get those from the bug reporter.

Also - if possible - it would be great if you could verify that those steps on Yakkety or later then succeed as expected since the upstream fix is already in there.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates

[Christian Ehrhardt  (paelzer)]

Gah obviously for vvfat it isn't an image to be created at all.
So I thought steps to reproduce are:

1. get a guest that works (via uvtoools-libvirt or whatever you prever)
2. get the qemu commandline that it is started with, in my case:
   $ sudo kvm -m 1024 -drive file=/dev/sdb,format=raw,cache=none,index=0,media=disk
3. create a dir (can be empty) to share in vvfat mode
   $ mkdir /tmp/sharevvfat
   $ touch /tmp/sharevvfat/hostfoo
4. start your guest with the path as vvfat set to share (ro for now)
   Append the following to your qemu cmdline:
     -drive format=vvfat,file=fat:/tmp/sharevvfat/,if=virtio
   check the block device content by mounting it and verifyinf that "hostfoo" is there
5. shutdown the guest to retry rw
6. For rw append it as:
     -drive format=vvfat,file=fat:rw:/tmp/sharevvfat/,if=virtio

I have echoed some content into a file, and while the update policy isn't instand after my guest shut down I could access on the host what the guest put there.
All worked fine, but it seems it was already failing in the background on guest shutdown I see the failing assertion and the guest data is not written correctly to the Host.

Retrying on a newer qemu than Xenial confirmed that there the same case is working.

[Christian Ehrhardt  (paelzer)]

If there are no blockers uncovered on the way I will merge this with the ongoing qemu SRU.
That means that a test builds will soon be available in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2697

[Christian Ehrhardt  (paelzer)]

Added SRU Template and tested 1:2.5+dfsg-5ubuntu10.13 from the bileto ppa.

Bug is fixed, content can be written from the guest and is re-readable from there as well as on the Host.

That said pushed to the SRU unapproved queue.

[zappacor (zappacor-l)]

Hi guys,

coming back from vacation last week, sorry for not replying before and.... tons of thanks for your prompt support and resolution!

BTW, by any chance, do you guys know what the status for a "bootsector=<path to file>" parameter for the VVFAT is?

Rolando

[Christian Ehrhardt  (paelzer)]

Hi,
I haven't touched on anything that would bring much more features to the vvfat code.
In general it is a helper to sync data, but it is brittle I'd not even so much like to see a bootsector support there. That I don't find anything doesn't mean there is nothing, if you know an upstream commit or discussion feel free to catch me, but if possible in a new bug or via IRC.

On this particular bug here we now wait for the SRU Team to accept that, the bug here on Lauchpad will get an update that it has to be verified to work (I'd totally appreciate your help there) and then it can be released into the stable release.

[zappacor (zappacor-l)]

Sure, no problem Christian!

BTW, I just updated my system and got this qemu version:
>>> QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.11), Copyright (c) 2003-2008 Fabrice Bellard
but the issue is still there:
>>> qemu-system-x86_64: /build/qemu-iZpOAh/qemu-2.5+dfsg/block/vvfat.c:2290: commit_direntries: Assertion `!strncmp(s->directory.pointer, "QEMU", 4)' failed.
>>> Aborted (core dumped)

However, downloading and running the patched version you mentioned above from:
>>> https://launchpadlibrarian.net/316972847/qemu-system-x86_2.5+dfsg-5ubuntu10.13_amd64.deb
does work perfectly!

Thanks Christian and don't hesitate to let me know once it gets accepted and included. I'll "apt-get update" my system and let you know the outcome.

Again, thanks,
Rolando

[Christian Ehrhardt  (paelzer)]

Subscribing Marc so he can easily track once this SRU passes.

[Robie Basak (racb)]

Hello Rolando, or anyone else affected,

Accepted qemu into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu10.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[zappacor (zappacor-l)]

Hi Robie,

I think something is wrong with the build you mention.

Still can't install it by means of:
>>> apt-get -s install qemu-system-x86/xenial-proposed
>>> apt-get -s install qemu-system/xenial-proposed
or
>>> apt-get -s install qemu-kvm/xenial-proposed
but issuing a:
>>> wget https://launchpad.net/ubuntu/xenial/amd64/qemu-system-x86/1:2.5+dfsg-5ubuntu10.13
gives me, after renaming it, this error:
>>> ./patched-qemu
>>> ./patched-qemu: line 1: syntax error near unexpected token `newline'
>>> ./patched-qemu: line 1: `<!DOCTYPE html>'
>>>
>>> ./patched-qemu --version
>>> ./patched-qemu: line 1: syntax error near unexpected token `newline'
>>> ./patched-qemu: line 1: `<!DOCTYPE html>'

Please note that I download *just* the qemu-system-x86 binary and nothing else than that.

HTH,
Rolando

[zappacor (zappacor-l)]

Hi guys,

sorry, my bad! Downloaded the wrong file, the correct one is:
>>> http://launchpadlibrarian.net/318154621/qemu-system-x86_2.5+dfsg-5ubuntu10.13_amd64.deb

So, I checked it and, yes, it works fine!

I got this errors but I think they are correct as I'm just pointing to the new binary instead of re-installing all the files for the package:
>>> Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-iscsi.so
>>> Note: only modules from the same build can be loaded.
>>> Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-curl.so
>>> Note: only modules from the same build can be loaded.
>>> Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so
>>> Note: only modules from the same build can be loaded.
>>> Failed to initialize module: /usr/lib/x86_64-linux-gnu/qemu/block-dmg.so
>>> Note: only modules from the same build can be loaded.

So, again, thank you guys very much!

I might be opening a new bug to request the backport of the boot record code (I need to find out where it was though...)

Thanks,
Rolando

[zappacor (zappacor-l)]

Hi guys,

unfortunatelly, I hit an issue with this patched version: there seems to be some issues, at least for LEAF (Linux Embedded Appliance Framework - https://sourceforge.net/projects/leaf).

Previous steps to reproduce (LEAF image creation):
 # This download is ~110M:
 wget https://downloads.sourceforge.net/project/leaf/Bering-uClibc/6.0.3/Bering-uClibc_6.0.3_x86_64_syslinux_serial115200.tar.gz
 qemu-img create -f qcow2 bering-own.qcow2 256M
 modprobe nbd
 qemu-nbd -c /dev/nbd0 bering-own.qcow2
 # Use fdisk to create a type c (FAT32 LBA) bootable partition like this:
 # Device Boot Start End Sectors Size Id Type
 # /dev/nbd0p1 * 2048 524287 522240 255M c W95 FAT32 (LBA)
 fdisk /dev/nbd0
 mkfs.vfat -n BERING -F 32 /dev/nbd0p1
 syslinux -i /dev/nbd0p1
 dd conv=notrunc bs=440 count=1 if=/usr/lib/SYSLINUX/mbr.bin of=/dev/nbd0
 mount /dev/nbd0p1 /mnt/
 tar -C /mnt -zxvf ./Bering-uClibc_6.0.3_x86_64_syslinux_serial115200.tar.gz

Not using the VVFAT it works OK:
 ./patched-qemu-christian-OK/qemu-system-x86_64\
  -nodefaults -nographic -localtime -enable-kvm -smp 1 -m 512 -serial stdio\
  -drive if=ide,file=bering-own.qcow2\
  -kernel /mnt/syslinux/linux -initrd /mnt/initrd.lrp\
  -append 'rw root=/dev/ram0 console=ttyS0,115200n8 VERBOSE=1 PKGPATH=/dev/sda1:vfat LRP=root,license,local'
 # Type all this to shutdown LEAF:
 # root + [ENTER] + [ENTER] + q + poweroff

However, when using the VVFAT emulation:
 ./patched-qemu-christian-OK/qemu-system-x86_64\
  -nodefaults -nographic -localtime -enable-kvm -smp 1 -m 512 -serial stdio\
  -drive if=ide,file=fat:rw:/mnt\
  -kernel /mnt/syslinux/linux -initrd /mnt/initrd.lrp\
  -append 'rw root=/dev/ram0 console=ttyS0,115200n8 VERBOSE=1 PKGPATH=/dev/sda1:vfat LRP=root,license,local'
LEAF displays these errors:
 LINUXRC: Mounted /dev/sda1 as vfat
 LINUXRC: Installing - root: dev: /dev/sda1 mnt: /mnt1 t: vfat f: /mnt1/root.lrp /dev/sda1gunzip: invalid magic
 (cpt!) license: dev: /dev/sda1 mnt: /mnt1 t: vfat f: /mnt1/license.lrp /dev/sda1gunzip: invalid magic
 (cpt!) local: dev: /dev/sda1 mnt: /mnt1 t: vfat f: /mnt1/local.lrp /dev/sda1gunzip: unexpected end of file
 (cpt!) configdb: dev: /dev/sda1 mnt: /mnt1 t: vfat f: /mnt1/configdb.lrp /dev/sda1gunzip: invalid magic
 (cpt!) - Finished.
 LINUXRC: Mounting squashfs with modules...
 mount: mounting /dev/loop0 on /lib/modules/4.4.61-x86_64 failed: Invalid argument
 LINUXRC: Squashfs mount failed!
 LINUXRC: loading modules from /etc/modules
 sed: /etc/modules: No such file or directory

To somehow debug it, I added the VVFAT as a second IDE HDD while using the qcow disk image as the primary boot drive:
 ./patched-qemu-christian-OK/qemu-system-x86_64\
  -nodefaults -nographic -localtime -enable-kvm -smp 1 -m 512 -serial stdio\
  -drive if=ide,file=bering-own.qcow2,index=0\
  -drive if=ide,file=fat:rw:/mnt\
  -kernel /mnt/syslinux/linux -initrd /mnt/initrd.lrp\
  -append 'rw root=/dev/ram0 console=ttyS0,115200n8 VERBOSE=1 PKGPATH=/dev/sda1:vfat LRP=root,license,local'
 # => root + {ENTER] + [ENTER] + q
Then, get errors when mounting the VVFAT disk partition (but it gets mounted...) as well as when trying to display a sub-dir of it:
 firewall# m...

[zappacor (zappacor-l)]

I meant "I'll be glad to help to have this solved" :-)

[zappacor (zappacor-l)]

Just in case, this the output of a "dmesg" command:

firewall# dmesg
[ 0.000000] Linux version 4.4.61-x86_64 (kapeka@stalker) (gcc version 5.3.0 (GCC) ) #1 SMP Thu Apr 13 19:03:07 CEST 2017
[ 0.000000] Command line: rw root=/dev/ram0 console=ttyS0,115200n8 VERBOSE=1 PKGPATH=/dev/sda1:vfat LRP=root,license,local
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] x86/fpu: Legacy x87 FPU detected.
[ 0.000000] x86/fpu: Using 'lazy' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001ffdffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000001ffe0000-0x000000001fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[ 0.000000] e820: last_pfn = 0x1ffe0 max_arch_pfn = 0x400000000
[ 0.000000] MTRR default type: write-back
[ 0.000000] MTRR fixed ranges enabled:
[ 0.000000] 00000-9FFFF write-back
[ 0.000000] A0000-BFFFF uncachable
[ 0.000000] C0000-FFFFF write-protect
[ 0.000000] MTRR variable ranges enabled:
[ 0.000000] 0 base 0080000000 mask FF80000000 uncachable
[ 0.000000] 1 disabled
[ 0.000000] 2 disabled
[ 0.000000] 3 disabled
[ 0.000000] 4 disabled
[ 0.000000] 5 disabled
[ 0.000000] 6 disabled
[ 0.000000] 7 disabled
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT
[ 0.000000] found SMP MP-table at [mem 0x000f6640-0x000f664f] mapped at [ffff8800000f6640]
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[ 0.000000] BRK [0x01c89000, 0x01c89fff] PGTABLE
[ 0.000000] BRK [0x01c8a000, 0x01c8afff] PGTABLE
[ 0.000000] BRK [0x01c8b000, 0x01c8bfff] PGTABLE
[ 0.000000] BRK [0x01c8c000, 0x01c8cfff] PGTABLE
[ 0.000000] RAMDISK: [mem 0x1ff57000-0x1ffdffff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F6460 000014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 0x000000001FFE16FA 000034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 0x000000001FFE0C14 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 0x000000001FFE0040 000BD4 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACS 0x000000001FFE0000 000040
[ 0.000000] ACPI: SSDT 0x000000001FFE0C88 0009C2 (v01 BOCHS BXPCSSDT 000...

[Christian Ehrhardt  (paelzer)]

Hi Rolando,
there are further changes in this build - we need to exclude that you are seeing this just due to only downloading the .deb file instead of enabling proposed to gain all required dependencies.

You can instead of an full "apt upgrade" also run rather selective upgrades.
I'd recommend
1. enabling proposed
2. call apt upgrade to check what is available but then abort
3. call apt install "<qemu packages you saw in step #2"
   # this will pull in all dependencies

[Christian Ehrhardt  (paelzer)]

Hi Rolando,
we really should stick to here verify the issue that we fixed (the vvfat rw fail) and after you gave me a 50%-heartache :-) I verified proposed on my own now.

It just works via
1. echo "deb http://archive.ubuntu.com/ubuntu/ xenial-proposed restricted main multiverse universe" >> /etc/apt/sources.list
2. apt update
3. apt install qemu-block-extra qemu-kvm qemu-system-common qemu-system-x86 qemu-utils

Now (re)-start your vvfat guest and it will work to write to the vvfat disk.
It did so for me at least.

I read through the issue you are seeing now and that is IMHO as a TL;DR saying "the vvfat emulation of a directory is not matching mounting it directly in the guest.
Am I right with that the summary of above is
1. create empty fat (not vvfat) image bering-own.qcow2
compare running
a) mount image to /mnt and append it as vvfat to the guest
vs
b) pass the qcow file itself as a drive to the guest
If I'm correct in reading what you do that is a totally different case.
If you look at the code [1] and some doc [2]+[3] you'll see that this driver is very restricted.
I'd never use it (or have seen it used) as root file system, not even for something critical other than simple data sharing between win guests and the host.
Quoting:
What you should never do:
- use non-ASCII filenames ;
- use "-snapshot" together with ":rw:" ;
- expect it to work when loadvm’ing ;
- write to the FAT directory on the host system while accessing it with the guest system.

There are slight improvements in [4], but also see the discussion [3] which ended as Won't Fix.

Quoting from [3] is good as well: "vfat is a last-resort quick-n-dirty fix for a problem when you have a guest which doesn't have any other ways to communicate except of a fat image (read: a floppy). In all other cases just don't use it."

[1]: https://github.com/qemu/qemu/blob/master/block/vvfat.c
[2]: https://en.wikibooks.org/wiki/QEMU/Devices/Storage
[3]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=419929
[4]: https://bugs.launchpad.net/qemu/+bug/1599539

All that said for the initially reported issue I can confirm that the bug is fixed and verified in proposed. Further insufficiencies of vfat would be another bug, but actually more upstream development work. If you open a new one I'm happy to discuss, but unlikely will find the time to do the related dev work.

[zappacor (zappacor-l)]

Hi Christian,

Ok, understood. And, yes, you are correct: basically I create a qcow image file and then:
- Bering works fine when passing qemu the qcow image file to be used as the HDD
- Bering doesn't work when mounting the same qcow image file and then launching qemu to access those same files from the mount point.

Please note that the issue happens while meeting all this conditions (already known by me):
  What you should never do:
  - use non-ASCII filenames ;
  - use "-snapshot" together with ":rw:" ;
  - expect it to work when loadvm’ing ;
  - write to the FAT directory on the host system while accessing it with the guest system.

Anyways, will open a new bug for this LEAF/Bering issue but, I agree with you, in that this feature should be used with caution, at least for now, as it seems to be still quite buggy.

Thanks Christian!

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.13

---------------
qemu (1:2.5+dfsg-5ubuntu10.13) xenial; urgency=medium

  * debian/patches/ubuntu/vvfat-fix-volume-name-assertion.patch:
    Fix the volume name assertion in vvfat rw mode (LP: #1684239)

qemu (1:2.5+dfsg-5ubuntu10.12) xenial; urgency=medium

  * debian/patches/ubuntu/bug-1656112-POWER8NVL-[12]-*.patch:
    Add PowerISA 2.07 compatibility mode to fix execution on POWER8NVL
    processors such as in S822LC (8335-GTB) machines (LP: #1656112)

 -- Christian Ehrhardt <email address hidden> Tue, 25 Apr 2017 13:58:10 +0200

[Andy Whitcroft (apw)]

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[zappacor (zappacor-l)]

Thank you all very much guys!

I'm on a business travel but will install update it as soon as I get back home.

[zappacor (zappacor-l)]

Hi Christian,

as stated above, found other bugs but as you said they seem to not to be related to this one so please feel free to close it.

Thanks,
Rolando.