smbd crash after reboot

[Svein Seldal]

Running Ubuntu 16.04.3 LTS, with samba 2:4.3.11+dfsg-0ubuntu0.16.04.11

My problem is that smbd stops after reboot. If I manually restart the smbd service, the service starts normally.

# systemctl status smbd
● smbd.service - LSB: start Samba SMB/CIFS daemon (smbd)
   Loaded: loaded (/etc/init.d/smbd; bad; vendor preset: enabled)
   Active: active (exited) since Fri 2017-11-17 21:57:16 CET; 1h 41min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 2250 ExecStart=/etc/init.d/smbd start (code=exited, status=0/SUCCESS)
    Tasks: 0
   Memory: 0B
      CPU: 0

Nov 17 21:57:12 neo systemd[1]: Starting LSB: start Samba SMB/CIFS daemon (smbd)...
Nov 17 21:57:15 neo smbd[2250]: * Starting SMB/CIFS daemon smbd
Nov 17 21:57:16 neo smbd[2250]: ...done.
Nov 17 21:57:16 neo systemd[1]: Started LSB: start Samba SMB/CIFS daemon (smbd).

/var/log/samba/log.smbd reveals that smbd has crashed

[2017/11/17 21:57:15.984660, 0] ../source3/lib/interface.c:543(load_interfaces)
  WARNING: no network interfaces found
[2017/11/17 21:57:16.209233, 0] ../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'smbd' finished starting up and ready to serve connections
[2017/11/17 21:57:16.209963, 0] ../source3/smbd/server.c:877(open_sockets_smbd)
  open_sockets_smbd: No sockets available to bind to.
[2017/11/17 21:57:16.210191, 0] ../source3/lib/util.c:789(smb_panic_s3)
  PANIC (pid 2263): open_sockets_smbd() failed
[2017/11/17 21:57:16.210643, 0] ../source3/lib/util.c:900(log_stack_trace)
  BACKTRACE: 9 stack frames:
   #0 /usr/lib/x86_64-linux-gnu/samba/libsmbregistry.so.0(log_stack_trace+0x1a) [0x7fc5520257aa]
   #1 /usr/lib/x86_64-linux-gnu/samba/libsmbregistry.so.0(smb_panic_s3+0x20) [0x7fc552025880]
   #2 /usr/lib/x86_64-linux-gnu/libsamba-util.so.0(smb_panic+0x2f) [0x7fc552d98f1f]
   #3 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0(+0x144461) [0x7fc552998461]
   #4 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base.so.0(+0x1447a1) [0x7fc5529987a1]
   #5 /usr/lib/x86_64-linux-gnu/samba/libsmbd-shim.so.0(exit_server+0x12) [0x7fc550872c52]
   #6 /usr/sbin/smbd(main+0x131e) [0x557e93538b1e]
   #7 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fc54f555830]
   #8 /usr/sbin/smbd(_start+0x29) [0x557e93539199]
[2017/11/17 21:57:16.210797, 0] ../source3/lib/util.c:801(smb_panic_s3)
  smb_panic(): calling panic action [/usr/share/samba/panic-action 2263]
[2017/11/17 21:57:16.212999, 0] ../source3/lib/util.c:809(smb_panic_s3)
  smb_panic(): action returned status 0
[2017/11/17 21:57:16.213070, 0] ../source3/lib/dumpcore.c:303(dump_core)
  dumping core in /var/log/samba/cores/smbd

(no core dump can be found at the indicated location unfortunately.)

With some aid in #ubuntu-server, this can probably be related to the sequencing of network coming up. This server use DHCP assigned server addresses, so there will be a delay between ifup and network being online.

As can be seen below: smbd seems to be starting much earlier than network online, while nmbd is started after network online.

Nov 17 21:57:11 neo kernel: r8169 0000:03:00.0 enp3s0: link up
Nov 17 21:57:11 neo kernel: IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
Nov 17 21:57:12 neo dhclient[2055]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 4 (xid=0x376d3044)
Nov 17 21:57:12 neo sh[2019]: DHCPDISCOVER on enp3s0 to 255.255.255.255 port 67 interval 4 (xid=0x376d3044)
Nov 17 21:57:12 neo dhclient[2055]: DHCPREQUEST of 10.5.5.10 on enp3s0 to 255.255.255.255 port 67 (xid=0x44306d37)
Nov 17 21:57:12 neo sh[2019]: DHCPREQUEST of 10.5.5.10 on enp3s0 to 255.255.255.255 port 67 (xid=0x44306d37)
Nov 17 21:57:12 neo sh[2019]: DHCPOFFER of 10.5.5.10 from 10.5.5.1
Nov 17 21:57:12 neo dhclient[2055]: DHCPOFFER of 10.5.5.10 from 10.5.5.1
Nov 17 21:57:12 neo dhclient[2055]: DHCPACK of 10.5.5.10 from 10.5.5.1
Nov 17 21:57:12 neo sh[2019]: DHCPACK of 10.5.5.10 from 10.5.5.1
Nov 17 21:57:12 neo systemd[1]: Starting LSB: start Samba SMB/CIFS daemon (smbd)...
Nov 17 21:57:15 neo smbd[2250]: * Starting SMB/CIFS daemon smbd
Nov 17 21:57:16 neo smbd[2250]: ...done.
Nov 17 21:57:16 neo systemd[1]: Started LSB: start Samba SMB/CIFS daemon (smbd).
Nov 17 21:57:17 neo dhclient[2055]: bound to 10.5.5.10 -- renewal in 33991 seconds.
Nov 17 21:57:17 neo sh[2019]: bound to 10.5.5.10 -- renewal in 33991 seconds.
Nov 17 21:57:18 neo systemd[1]: Started Raise network interfaces.
Nov 17 21:57:18 neo systemd[1]: Reached target Network.
Nov 17 21:57:18 neo systemd[1]: Starting A high performance web server and a reverse proxy server...
Nov 17 21:57:18 neo systemd[1]: Starting Statistics collection and monitoring daemon...
Nov 17 21:57:18 neo systemd[1]: Starting OpenBSD Secure Shell server...
Nov 17 21:57:18 neo systemd[1]: Starting Docker Application Container Engine...
Nov 17 21:57:18 neo systemd[1]: Started Unattended Upgrades Shutdown.
Nov 17 21:57:18 neo systemd[1]: Reached target Network is Online.
Nov 17 21:57:18 neo systemd[1]: Started Daily apt download activities.
Nov 17 21:57:18 neo systemd[1]: Started Daily apt upgrade and clean activities.
Nov 17 21:57:18 neo systemd[1]: Reached target Timers.
Nov 17 21:57:18 neo systemd[1]: Starting /etc/rc.local Compatibility...
Nov 17 21:57:18 neo systemd[1]: Starting LSB: start Samba NetBIOS nameserver (nmbd)...
Nov 17 21:57:18 neo systemd[1]: Starting LSB: start Samba daemons for the AD DC...
Nov 17 21:57:18 neo systemd[1]: Starting LSB: disk temperature monitoring daemon...
Nov 17 21:57:18 neo systemd[1]: Started LSB: Postfix Mail Transport Agent.
Nov 17 21:57:18 neo systemd[1]: Started /etc/rc.local Compatibility.
Nov 17 21:57:18 neo systemd[1]: Started LSB: Network share OpenZFS datasets..

Interestingly, smbd seems to depend on nmbd, yet they are started in the oposite order.

# systemd-analyze critical-chain smbd.service
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

smbd.service +3.403s
└─nmbd.service @24.357s +471ms
  └─network-online.target @24.227s
    └─network.target @24.084s
      └─networking.service @13.869s +10.212s
        └─apparmor.service @12.893s +948ms
          └─local-fs.target @12.885s
            └─var-lib-docker-aufs.mount @29.826s
              └─var-lib-docker.mount @12.746s
                └─local-fs-pre.target @5.508s
                  └─systemd-tmpfiles-setup-dev.service @2.679s +1.193s
                    └─kmod-static-nodes.service @2.461s +198ms
                      └─systemd-journald.socket @2.366s
                        └─-.slice @2.181s

[Andreas Hasenack (ahasenack)]

Thanks for filing this bug in Ubuntu.

Could you please attach your /etc/samba/smb.conf file?

I would also like to take a look at the log files from /var/log/samba/log*

Thanks

[Svein Seldal (sveinse)]

Sure, but do you need *everything* from /var/log/samba/log* ? This is running on a production server, and I've got 63 files in this directory. I need to wash them to ensure I don't publish any sensitive data.

[Andreas Hasenack (ahasenack)]

log.smbd and log.nmbd would be enough to start with. For now I believe that smb.conf has an interfaces line specifying a network or NIC that doesn't exist at boot time, but only after you login.

[Launchpad Janitor (janitor)]

[Expired for samba (Ubuntu) because there has been no activity for 60 days.]

[Svein Seldal (sveinse)]

Took a while before the server was rebooted. Attached log.nmbd, log.smbd and smbd.conf

[Andreas Hasenack (ahasenack)]

smb.conf:
 bind interfaces only = Yes
 dns proxy = No
 interfaces = 10.5.5.0/24

log.nmbd:
[2018/04/04 14:05:08.221963, 0] ../source3/lib/interface.c:543(load_interfaces)
  WARNING: no network interfaces found
  WARNING: no network interfaces found
[2018/04/04 14:05:08.424684, 0] ../lib/util/become_daemon.c:124(daemon_ready)
  STATUS=daemon 'smbd' finished starting up and ready to serve connections
[2018/04/04 14:05:08.425310, 0] ../source3/smbd/server.c:877(open_sockets_smbd)
  open_sockets_smbd: No sockets available to bind to.
[2018/04/04 14:05:08.425524, 0] ../source3/lib/util.c:789(smb_panic_s3)
  PANIC (pid 2283): open_sockets_smbd() failed

How is the networking on this machine setup? Specifically, the interface(s) on 10.5.5.0/24. It looks like samba (nmbd specifically) is starting before that interface is up. Is this a desktop perhaps, with a wifi connection?

Can you share /etc/network/interfaces and /etc/network/interfaces.d/* ?

[Launchpad Janitor (janitor)]

[Expired for samba (Ubuntu) because there has been no activity for 60 days.]