Ubuntu-16.04-amd64 - booting via NFS using overlayroot fails

[Goran]

I have a setup where I need to boot hundreds of diskless clients with Ubuntu 16.04 via one NFS-server with one NFS-share.
Therefore I need a readonly-NFS-share which will be mounted by the clients with the help of an overlay-fs in readwrite-mode.

This exact setup was already achieved by me with other distributions but not with Ubuntu-16.04-amd64.

My setup is following:

  - dhcp-server
  - tftp-server (bootloader, vmlinuz, initrd)
  - nfs-server (root file system)
  - diskless-client

I installed Ubuntu-16.04-amd64 onto the nfs-server with debootstrap under /srv/nfs/xenial. I chrooted into this directory and installed linux-image-generic and overlayroot. After installation I edited /etc/overlayroot.conf:

    overlayroot="tmpfs"

and created a new initrd. The kernel and new initrd went to the tftp-server-directory.

After I started the diskless-client with the help of the bootloader:

    KERNEL /xenial/vmlinuz-4.4.0-131-generic
    APPEND initrd=/xenial/initrd.img-4.4.0-131-generic root=/dev/nfs nfsroot=172.17.0.1:/mnt,ro ip=dhcp console=tty1 console=ttyS0,115200n8

Now vmlinuz and initrd are loaded and starting the boot-process. Below the important part of the kernel-messages.

.
.
.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/nfs-top ... done.
Begin: Running /scripts/nfs-premount ... done.
[ 4.655524] FS-Cache: Loaded
[ 4.692919] RPC: Registered named UNIX socket transport module.
[ 4.698926] RPC: Registered udp transport module.
[ 4.703435] RPC: Registered tcp transport module.
[ 4.705922] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 4.752465] FS-Cache: Netfs 'nfs' registered for caching
IP-Config: ens3 hardware address 52:54:00:89:cd:39 mtu 1500 DHCP
IP-Config: no response after 2 secs - giving up
IP-Config: ens3 hardware address 52:54:00:89:cd:39 mtu 1500 DHCP
IP-Config: ens3 guessed broadcast address 172.17.255.255
IP-Config: ens3 complete (dhcp from 172.17.0.1):
 address: 172.17.1.1 broadcast: 172.17.255.255 netmask: 255.255.0.0
 gateway: 0.0.0.0 dns0 : 0.0.0.0 dns1 : 0.0.0.0
 rootserver: 172.17.0.1 rootpath: /
 filename : lpxelinux.0
NFS over TCP not available from 172.17.0.1
Begin: Retrying nfs mount ... NFS over TCP not available from 172.17.0.1
done.
Begin: Retrying nfs mount ... NFS over TCP not available from 172.17.0.1
done.
Begin: Retrying nfs mount ... NFS over TCP not available from 172.17.0.1
done.
Begin: Retrying nfs mount ... NFS over TCP not available from 172.17.0.1
done.
Begin: Retrying nfs mount ... done.
done.
Begin: Running /scripts/nfs-bottom ... done.
Begin: Running /scripts/init-bottom ... Warning: overlayroot: configuring overlayroot with driver=overlay mode=tmpfs opts='' per /dev/nfs/etc/overlayroot.conf
/scripts/init-bottom/overlayroot: line 790: can't create /root/etc/fstab: Permission denied
Failure: overlayroot: failed to modify /etc/fstab (step 1)
/scripts/init-bottom/overlayroot: line 805: can't create /root/etc/fstab: Permission denied
Failure: overlayroot: failed to modify /etc/fstab (step 2)
Success: overlayroot: configured root with 'tmpfs' using overlay per /dev/nfs/etc/overlayroot.conf
done.
[ 21.340149] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[ 21.349077] systemd[1]: Detected virtualization qemu.
[ 21.354666] systemd[1]: Detected architecture x86-64.

Welcome to Ubuntu 16.04.5 LTS!
.
.
.

After kernel messages the login prompt appears, but when entering the username and password, no authentication can be done.

Ubuntu 16.04.5 LTS nfs-image ttyS0

nfs-image login: g
Passwort:

Login incorrect

I repeated this setup with a debootstrapped root-file-system as well as one created with ordinary installer. Both same results. Further I installed SSH-server to login remotly but neither this worked. I can't see inside the machien what is happening so it's unlikely to find a solution this way.

[Goran (taketwo)]

For completeness I attach the full kernel log:

Loading /xenial/vmlinuz-4.4.0-131-generic... ok
Loading /xenial/initrd.img-4.4.0-131-generic...ok
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-131-generic (buildd@lgw01-amd64-015) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.10) ) #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 (Ubuntu 4.4.0-131.157-generic 4.4.134)
[ 0.000000] Command line: BOOT_IMAGE=/xenial/vmlinuz-4.4.0-131-generic initrd=/xenial/initrd.img-4.4.0-131-generic root=/dev/nfs nfsroot=172.17.0.1:/mnt ip=dhcp console=tty1 console=ttyS0,115200n8
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] x86/fpu: Legacy x87 FPU detected.
[ 0.000000] x86/fpu: Using 'eager' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003ffdffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000003ffe0000-0x000000003fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] Hypervisor detected: KVM
[ 0.000000] e820: last_pfn = 0x3ffe0 max_arch_pfn = 0x400000000
[ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT
[ 0.000000] found SMP MP-table at [mem 0x000f6a60-0x000f6a6f] mapped at [ffff8800000f6a60]
[ 0.000000] Scanning 1 areas for low memory corruption
[ 0.000000] RAMDISK: [mem 0x3d8d4000-0x3feb9fff]
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x00000000000F6880 000014 (v00 BOCHS )
[ 0.000000] ACPI: RSDT 0x000000003FFE1551 00002C (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACP 0x000000003FFE13E5 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.000000] ACPI: DSDT 0x000000003FFE0040 0013A5 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.000000] ACPI: FACS 0x000000003FFE0000 000040
[ 0.000000] ACPI: APIC 0x000000003FFE14D9 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.000000] No NUMA configuration found
[ 0.000000] Faking a node at [mem 0x0000000000000000-0x000000003ffdffff]
[ 0.000000] NODE_DATA(0) allocated [mem 0x3ffdb000-0x3ffdffff]
[ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[ 0.000000] kvm-clock: cpu 0, msr 0:3ffd3001, primary cpu clock
[ 0.000000] kvm-clock: using sched offset of 696260153027 cycles
[ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[ 0.000000] Zone ranges:
[ 0.000000] DMA [mem 0x0000000000001000-0x00000...

[Scott Moser (smoser)]

Hi,
Thanks for the bug report.

It sure looks like everything is set up correctly. There are no errors to the console until it fails to write to /root/etc/fstab. We test overlayroot with ubuntu kernels daily, but not with nfs as the underlying filesystem. overlayfs and nfs likely aren't well tested together. You could test with a kernel package 'linux-image-generic-hwe-16.04'. That would get you at 4.15 kernel.

While it is not likely in the path at this point, apparmor also doesn't get along terribly well with overlayfs. so you might need to pass apparmor=0.

Please put the following in your /etc/overlayroot.conf:
   overlayroot="tmpfs,debug=1"

That will get us some more information on the console. Then, please post that whole console log.

Unfortunately I do not think it will get us enough info.

You may end up booting with 'debug' on the kernel command line. That will get you a slew of 'set -x' output.

[Scott Moser (smoser)]

I received in email from Goran the following:

> Untested on Xenial but as far as it concerns other versions and other
> distributions as well, the only solution will be to install a kernel
> 4.6 or newer.
>
> Here is a link with the description.
>
> https://www.systutorials.com/linux-kernels/617530/ovl-default-permissions-linux-4-5/
>
> Goran Vukoman

So I'm going to mark this as "Won't Fix", but really its more "Can't fix" for
overlayroot.

@Goran,
Thank you for following up.