ISO Mint 17.2 live: ERROR (initramfs) /cow format specified as aufs and no support found.

[Cyberbob]

Hi Kendek.
first of all THANKS for your job.
Systemback a a fantastic utility for me!!

Ok,
with MINT 17.2 Rafaela 64 bit, MATE 1.10 kernel 3.16.0-58, me and other peoples on the italian forum, are reporting that are having this error, just after starting to load of ISO Live system from USB pen drive, systemback latest version:
1.6.201_08.26.2015_Qt5.2_GCC4.8.4_amd64.

ERROR we receive is:

Busybox V 1.21.1 (Ubuntu 1:1.21.0-1ubuntu1) buil-in shell (ash)
enter 'help' for a list of built-in comands.

(initramfs) /cow format specified as aufs and no support found.

What can you tell me about.
At you disposal for more info.
Thanks

[Kendek (nemh)]

I reproduced this problem, so I will try to find the cause.

[Cyberbob (fabiocasi)]

Hi Kendek.

Probably the problem is generated by CASPER that in the broken system is version 1.340.2 and in the good system is just 1.340!!!

[Kendek (nemh)]

Yeah, the problem does not occur with the Casper version 1.340. But when I tried with Ubuntu Trusty and Casper 1.340.2, then the problem was also gone. So the Ubuntu is good but the Mint is not. I tried with same kernel and checked the files, but something is wrong with the Mint.
But it is sure this problem is not a Systemback bug. So I cannot solve this in the Systemback.

[Thomas Weissel (xapient)]

according to this thread http://ubuntuforums.org/showthread.php?t=2299959
it sounds like aufs module is just not correctly included and there is a solution possible through the live system creation process...

did you find any other clue on why this is happening and how we can work around this problem?

i just tried systemback with kubuntu 16.04 and it drops to a busybox with the message

"/cow format specified as aufs and no support found"

just before that it said "/init line7 cant open /dev/sr0 no medium found"

[Thomas Weissel (xapient)]

you somewhere recommended downgrading casper to 1.340 to avoid this problem

the current version of casper is 1.376 in "kubuntu 16.04" and a downgrade is not possible anymore.. it conflicts with localechoser-data (which is needed for systemback) and it needs libplymouth2 (which is beeing replaced by "plymouth"

what exactly is the problem with casper (the difference to the newer versions)

could we patch casper somehow to make it work again?

[Thomas Weissel (xapient)]

i installed the package "unionfs-fuse" and now i get a different error ^^

[Kendek (nemh)]

Xapient, I tried to reproduce your problem with the Kubuntu 16.04 amd64, but I failed. I tried in Virtualbox and with pendrive (on a real machine), with BIOS (CSM) and UEFI, but in any case it was good.
The original issue was a Casper problem only with the Linux Mint 17, the Ubuntu 14.04 was not affected.
What kernel you using? Maybe the problem source is the kernel that is different from the default (currently, 4.4.0-21-generic).

[Thomas Weissel (xapient)]

thank you for your answer...

i am using the exact same kernel..
i am browsing the web all day already - searching for a solution.. that's where someone pointed me to "unionfs-fuse" and well.. it solved the first error.... (do you have it installed?)

i tried 2 different computers.. a netbook and my workstation (both running kubuntu 16.04 and up2date)

i also tried to create an ISO file and i used unetbootin to copy the file to the flashdrive and
i tried a different flashdrive because i read somewhere the flashdrive could be the cause..

:-(

shouldn't the error say.. can not mount filesystem.squashfs on /dev/loop ?? (and not the other way around?)

oh.. on my workstation i am using the #neon testing repository... but this shouldn't be a problem because they only deliver kde specific updates

[Thomas Weissel (xapient)]

oh.. and i am using

Package: systemback
Version: 1.8.999+bzr326~ubuntu16.04.1

because it didn't work with the stable version..
would you recommend the stable version for further testing ?

thx

[Kendek (nemh)]

I tested the Ubuntu (running on my HP ProBook 455 G1) and Kubuntu (running in VirtualBox) 16.04 (both up2date) with same Systemback version, Casper 1.376 and kernel 4.4.0-21-generic.
The unionfs-fuse is a (slow) backup solution, used when no overlay, overlayfs, aufs or unionfs support in the kernel. But the Ubuntu kernel contains the aufs (default in the oldest Ubuntu releases) and the overlay (default in the 16.04 and this is the official solution in the vanilla kernel), see the following command output:

cat /boot/config-$(uname -r) | grep -e AUFS -e OVERLAY

Both are in the kernel in module format. If you wish to choose between them, just edit the kernel parameters (TAB with Syslinux and 'E' with UEFI):

UNIONFS=overlay
UNIONFS=aufs
UNIONFS=unionfs-fuse

See these screenshots:
http://logout.hu/dl/upc/2016-05/180556_set.png
http://logout.hu/dl/upc/2016-05/180556_aufs.png
http://logout.hu/dl/upc/2016-05/180556_overlay.png
http://logout.hu/dl/upc/2016-05/180556_unionfs-fuse.png

As you see I do not found any problem.
Currently the Systemback 1.8.999 contains a little debug fix, but otherwise equal with the stable version (1.8.300). So the plus tests with the stable version is not necessary.

[Thomas Weissel (xapient)]

oke.. so i "somehow" managed to get the flashdrive to boot - i'm really sorry.. i don't know what i changed on the system.. also only the netbook system is able to create a bootable flashdrive..

i immediately checked / and it is unionfs-fuse

so i used the kernel parameter and tried aufs, overlay and unionfs and every one of them dropped me into a busibox..

"/cow format specified as aufs and no support found" (replace aufs with overlay or unionfs)

so even if i have the exact same kernel it seems aufs or overlaymodules are not available..
how can i check that on the running (original) system and probably add it to the system..

(i would be really happy with unionfs-fuse but since you said its slow ^^ - also i would really like to know whats wrong here )

thx again

[Thomas Weissel (xapient)]

at the original system

locate aufs
/lib/modules/4.4.0-21-generic/kernel/fs/aufs
/lib/modules/4.4.0-21-generic/kernel/fs/aufs/aufs.ko

[Kendek (nemh)]

I tried to reinstall the 16.04 (because my installed system is upgraded from the 15.10 and my test installations are also upgraded from the 16.04 beta releases). I created new Live image and tried to boot. But I did not experience any change, the Live has started normally.
I am sorry, but if I cannot reproduce the problem then I cannot fix it (or suggest a workaround, only the system reinstallation).

[Thomas Weissel (xapient)]

well thank you anyway.. if it is okay (before i try a complete reinstallation i would ask you one or two further questions..

i just checked that linux-image-extra-4.4.0-21-generic is installed and then i checked if overlayfs or aufs are loaded..

lsmod|grep overlay

they are NOT loaded... but available..

modprobe overlayfs
lsmod |grep overlay
overlay 49152 0

is it mandatory to have those modules loaded for them to be included in the initramfs of the live system??

i narrowed it down to "why is overlayfs or aufs not available in the live systems ramdisk?"

modprobe overlayfs works fine on the live system so it is included somewhere - but not at boot time.

is it correct that the problem is in the ramdisk then or is this a missconception of mine?

thank you in advance.

[Thomas Weissel (xapient)]

i think i just found a solution to this problem !

i added

overlayfs
aufs

to

/etc/initramfs-tools/modules

and ran

sudo update-initramfs -u

after a reboot lsmod showed that both modules were loaded.
i then started systemback and created a new livesystem..
running "mount" on the livesystem showed that / is using overlayfs

success?

i give it another try on my target system (the netbook) to be sure :-)

[Kendek (nemh)]

Ok, this is a workaround (I think). But now I using a very clear (installed today) system (Ubuntu 16.04), and I cheked the /boot/initrd.img-4.4.0-22-generic file (this is just a compressed directory structure), and contains the following modules:
lib/modules/4.4.0-22-generic/kernel/fs/aufs/aufs.ko
lib/modules/4.4.0-22-generic/kernel/fs/overlayfs/overlay.ko
and the following executable:
usr/bin/unionfs
(plus bin/unionfs-fuse symlink).
These because the /usr/share/initramfs-tools/hooks/casper, when updating the initrd.img. So every file is in place. I wonder why that is not the case with you.

[Ray Sherwin (dark-sky)]

Well I think you are on the right track with your snooping. I discovered last night that my kernel 4.18 does not have aufs filesystem modules compiled in. Linus has stopped putting it in his kernel. I went to here and at the present recompiling my 4.18.0-rc2+ kernel with make deb-pkg using the patches only to patch my existing 4.18.0-rc2+ kernel source tree following the README in the git directory for aufs4-standalone tree section:

"For aufs4-standalone tree,
There are several ways to build.

1.
- apply ./aufs4-kbuild.patch to your kernel source files.
- apply ./aufs4-base.patch too.
- apply ./aufs4-mmap.patch too.
- apply ./aufs4-standalone.patch too, if you have a plan to set"

I also applied vfs-ino.patch & tmpfs-idr.patch but held off on aufs4-loopback.patch as the directions says it is not needed unless you run into a error message.

My kernel .config file now looks like this with the aufs patches applied using original distro kernel as a guide:

CONFIG_AUFS_FS=y
CONFIG_AUFS_BRANCH_MAX_127=y
# CONFIG_AUFS_BRANCH_MAX_511 is not set
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
CONFIG_AUFS_EXPORT=y
CONFIG_AUFS_INO_T_64=y
CONFIG_AUFS_XATTR=y
# CONFIG_AUFS_FHSM is not set
# CONFIG_AUFS_RDU is not set
CONFIG_AUFS_DIRREN=y
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BR_HFSPLUS=y
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set

The git aufs repo is here that I used for the aufs4-standalone tree:

https://github.com/sfjro/aufs4-standalone

[Ray Sherwin (dark-sky)]

I will post my results when the deb packages are built and applied / tested with my image here I am building; although I am not using Systemback I seem to be having the same issue as you with Cubic and other methods I have been trying over the last few days.

Fingers crossed here....

[Ray Sherwin (dark-sky)]

It does not seem to be using fuse from the config from the kernel that came with my live cd but I could be wrong:

# CONFIG_AUFS_BR_FUSE is not set

[Ray Sherwin (dark-sky)]

I did match what the original kernel fuse had in the live install kernel .config I forgot to mention just in case:

CONFIG_FUSE_FS=y
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=y
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set

The "# CONFIG_OVERLAY_FS_XINO_AUTO is not set" line was not in the original kernel .config (it may be a new feature) so I left it as default as "is not set" for the time being.

[mk (mk.pmb)]

I solved a similar problem today, same error message but with "overlayfs" instead of "aufs". In my case the "no support found" was misleading so there's a chance this problem here isn't about aufs either.

In my case the problem turned out to be that I used an outdated folder structure inside my casper-rw partition file. Up until Ubuntu Xenial, I could just put my /etc and /home modifications into the root (top level) directory of my CRW file.

I noticed that Ubuntu Bionic would boot if I rename my CRW file so it wouldn't be detected. Thus I knew the defective part was my CRW file.
I created a new CRW file holding just a fresh blank ext4 file system and tried that. It worked, the live session booted. I shut that down and inspected my CRW file. It held lots of files created by the live session, like the /var/syslog. The difference was, they weren't on the top level (/mnt/casper-rw/var/syslog) but in a directory "upper" (/mnt/casper-rw/upper/var/syslog) and casper had created some other directories on the top level.

So I restored my old CRW file, mounted it, created a folder "upper" and moved my home and etc directories into that "upper". The only other directory remaining on the top level was "lost+found". I booted with this and it worked. All my old casper data was available in the live session.

Maybe it works for your casper as well, even with aufs.

[mk (mk.pmb)]

Of course the syslog was /var/log/syslog.