snapd does not parse /etc/fstab properly when using mhddfs

[Ibrahim Yashau]

If you put in any mhddfs mount lines in /etc/fstab such as the one below, snapd.service fails to start.

mhddfs# /mnt/dir1,/mnt/dir2 /mnt/dir fuse defaults,allow_other 0 0

It was working fine before and it started with version 2.32+18.04

Apr 03 13:43:33 asd snapd[4016802]: AppArmor status: apparmor is enabled and all features are available
Apr 03 13:43:33 asd snapd[4016802]: 2018/04/03 13:43:33.377173 backend.go:104: cannot determine if NFS is in use: cannot parse /etc/fstab: expected between 3 and 6 fields, found 1
Apr 03 13:43:33 asd snapd[4016802]: 2018/04/03 13:43:33.378091 system_key.go:117: cannot determine nfs usage in generateSystemKey: cannot parse /etc/fstab: expected between 3 and 6 fields, found 1
Apr 03 13:43:33 asd snapd[4016802]: 2018/04/03 13:43:33.378103 helpers.go:115: error trying to compare the snap system key: cannot parse /etc/fstab: expected between 3 and 6 fields, found 1
Apr 03 13:43:33 asd snapd[4016802]: 2018/04/03 13:43:33.378649 system_key.go:117: cannot determine nfs usage in generateSystemKey: cannot parse /etc/fstab: expected between 3 and 6 fields, found 1
Apr 03 13:43:33 asd snapd[4016802]: error: cannot parse /etc/fstab: expected between 3 and 6 fields, found 1
Apr 03 13:43:33 asd systemd[1]: snapd.service: Main process exited, code=exited, status=1/FAILURE
Apr 03 13:43:33 asd systemd[1]: snapd.service: Failed with result 'exit-code'.
-- Subject: Unit snapd.service has failed
-- Unit snapd.service has failed.
Apr 03 13:43:33 asd systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
Apr 03 13:43:33 asd systemd[1]: snapd.service: Scheduled restart job, restart counter is at 5.
-- Automatic restarting of the unit snapd.service has been scheduled, as the result for
-- Subject: Unit snapd.service has finished shutting down
-- Unit snapd.service has finished shutting down.
Apr 03 13:43:33 asd systemd[1]: snapd.service: Start request repeated too quickly.
Apr 03 13:43:33 asd systemd[1]: snapd.service: Failed with result 'exit-code'.
-- Subject: Unit snapd.service has failed
-- Unit snapd.service has failed.
Apr 03 13:43:33 asd systemd[1]: snapd.socket: Failed with result 'service-start-limit-hit'.

I have removed snapd for the time being until this is resolved.

[Zygmunt Krynicki (zyga)]

Thank you for reporting the bug.

Are you saying the actual line contains the string:

mhddfs# /mnt/dir1,/mnt/dir2 /mnt/dir fuse defaults,allow_other 0 0

With the "mhddfs#" in the first column, as the device type?

[Zygmunt Krynicki (zyga)]

Dear reporter, can you please attach your entire fstab? While snapd should not refuse to run, it looks like your fstab has a line with just one field in it. That or our parser is terribly broken. In either case I'd like to see.

Based on your input so far I added a quick test case and it is parsed correctly, so it must not be it:

+ e, err = osutil.ParseMountEntry("/mnt/dir1,/mnt/dir2 /mnt/dir fuse defaults,allow_other 0 0")
+ c.Assert(err, IsNil)
+ c.Assert(e.Name, Equals, "/mnt/dir1,/mnt/dir2")
+ c.Assert(e.Dir, Equals, "/mnt/dir")
+ c.Assert(e.Type, Equals, "fuse")
+ c.Assert(e.Options, DeepEquals, []string{"defaults", "allow_other"})
+ c.Assert(e.DumpFrequency, Equals, 0)
+ c.Assert(e.CheckPassNumber, Equals, 0)

[Zygmunt Krynicki (zyga)]

I'm marking this as critical because regardless of the fstab snapd should never refuse to start up.

[Ibrahim Yashau (yashau)]

My /etc/fstab is fine. It has been like this forever. And yes, the line is supposed to start with mhddfs# for mounts dealing with it. I'm not sure if the problem is mhddfs having an odd syntax for the fstab mount or if it is your parser. Either way, it used to work fine before.

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
UUID=b0b2b411-3efb-498f-9e23-6a2a5ec8b704 / ext4 errors=remount-ro 0 1

UUID=cd0f0426-31f0-4eaf-885f-923041f8857b /mnt/local ext4 noatime,nobarrier,data=writeback 0 2

/swapfile none swap sw 0 0

mhddfs# /mnt/local,/media/external /mnt/merged fuse defaults,allow_other 0 0

[Zygmunt Krynicki (zyga)]

I understand that problem now, it is the # character that we mistake as a comment field. I will adjust the parser and we will fix snapd not to fail in this situation. Thank you for reporting this!

[Zygmunt Krynicki (zyga)]

Hmm

I'm kind of puzzled by how to parse this. It looks like the first field should be parsed as "mhddfs# /mnt/local,/media/external". Technically the space character must be escaped, but isn't. I need to do some digging as reality trumps specs, apparently.

[Michael Vogt (mvo)]

https://github.com/snapcore/snapd/pull/4969

[Zygmunt Krynicki (zyga)]

Dear reporter, can you please check one more thing. According to https://romanrm.net/mhddfs the syntax is slightly different:

mhddfs#/mnt/hdd1,/mnt/hdd2,/mnt/hdd3 /mnt/virtual fuse defaults,allow_other 0 0

Note that there is no more space between the mhddfs# and the initial mount point. This is in line with the specification of how fstab files must look like. Is the mount line with the space even working for you?

[Ibrahim Yashau (yashau)]

I'm sorry, I'm not sure how the space got there. I will edit it.

[Ibrahim Yashau (yashau)]

Nvm I cannot edit it.

[Ibrahim Yashau (yashau)]

I have done some testing and snapd refuses to start with or without that space. My fstab does not have it though.

[Zygmunt Krynicki (zyga)]

Yes, that's right. I will fix the issue when the space is not there. We will release the fix shortly. I was just trying to say that I can make it work without the space (then this is in agreement with the specification).

[Zygmunt Krynicki (zyga)]

This is now fixed in master and will be coming out in the next point release, 2.32.3

[Chris Halse Rogers (raof)]

Is there a way for me to test this locally? My setup (with a multi-device bcachefs filesystem) also fails here, and I'd like to check that this fixes my setup too.

[John Lenton (chipaca)]

@raof @zyga says it's in 2.32.3, so you should have it if your core has refreshed (check 'snap version').

[Chris Halse Rogers (raof)]

@chipaca: My core has refreshed, but I had snapd pinned to 2.32+18.04~pre6. After removing the apt pin and updating to snapd 2.32+18.04 snapd fails to start, so I guess this also needs a snapd release?

[Ibrahim Yashau (yashau)]

@raof I had to do snap refresh --candidate core

[Chris Halse Rogers (raof)]

@yashau: I cannot use any snap commands; they all fail, because snapd isn't running.

[Chris Halse Rogers (raof)]

For the record:
 ✘ ⚙ chris@Extravaganza  ~/Canonical/Mir/mir   logind-device-suspend-resume ●  snap refresh --candidate core
error: cannot communicate with server: Post http://localhost/v2/snaps/core: dial unix /run/snapd.socket: connect: connection refused

[Ibrahim Yashau (yashau)]

@raof You can edit your fstab temporarily to make snapd run. It won't change any of your mounts just editing the file.

[Chris Halse Rogers (raof)]

Ah, no, I can't. My problem is snapd fails parsing /proc/self/mountinfo! I guess I need to file a different bug.

[Zygmunt Krynicki (zyga)]

Oh boy, I should have seen that coming. Yes please file a separate bug and I'll get right to fixing it.

[Markus Birth (mbirth)]

Seems to work now. Just edited out my sshfs# and curlftpfs# entries from fstab, started snapd, refreshed core and now it seems to stop/start just fine even with the entries in fstab activated again.

[Tim Passingham (tim-8aw3u04umo)]

I seem to have a similar problem and have very slow boot times on 18.04. The snapd errors are:
May 23 12:21:53 snapd[1938]: 2018/05/23 12:21:53.075147 backend.go:104: cannot determine if NFS is in use: cannot parse /etc/fstab: expected between 3 and 6 fields, found 2
May 23 12:21:53 snapd[1938]: 2018/05/23 12:21:53.098248 system_key.go:118: cannot determine nfs usage in generateSystemKey: cannot parse /etc/fstab: expected between 3 and 6 fields, found 2
May 23 12:21:53 snapd[1938]: 2018/05/23 12:21:53.098336 helpers.go:115: error trying to compare the snap system key: cannot parse /etc/fstab: expected between 3 and 6 fields, found 2

my fstab is:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# / was on /dev/sda7 during installation
UUID=26455497-348c-470c-9055-d079a1ba482e / ext4 errors=remount
-ro 0 1
# /boot/efi was on /dev/sda2 during installation
UUID=0882-213D /boot/efi vfat umask=0077 0 1
# /home was on /dev/sda8 during installation
UUID=06550f02-a6d6-4077-b0ad-66a4e198bd58 /home ext4 defaults
  0 2
# swap was on /dev/sda9 during installation
UUID=d9fcd069-3ba9-41e8-a481-f6d37cda0d72 none swap sw
# Windows - Win 10 faststart must be disabled to get write access to TimWin
/dev/disk/by-uuid/EA3E2E083E2DCE7F /mnt/EA3E2E083E2DCE7F auto nosuid,nodev,nofai
l,noauto 0 0
/dev/disk/by-uuid/3084-647D /mnt/3084-647D auto nosuid,nodev,nofail,noauto 0 0
/dev/disk/by-uuid/8E3A496E3A49547D /mnt/8E3A496E3A49547D auto nosuid,nodev,nofai
l,noauto 0 0
/dev/disk/by-uuid/E0EE8023EE7FF060 /mnt/E0EE8023EE7FF060 auto nosuid,nodev,nofai
l,noauto 0 0
UUID=7C6A833C6A82F266 /mnt/TimWin ntfs defaults
  0 2