Samba panic or segfault

[Marc Vanhoomissen]

Once a day, we are confronted to 'samba panic or segfault' when trying to access samba files. The client computer keeps asking for password but rejects every password. At that point, access to files becomes impossible on that PC and very slow on all others. The only solution is to restart the samba servers.

At that point, the logs are full of lines like (100 kb for each PC):
[2019/03/27 13:01:58.639514, 0] ../source3/smbd/smbXsrv_session.c:1402(smbXsrv_session_update)
  smbXsrv_session_update: global_id (0x9506dc8a) store failed - NT_STATUS_UNSUCCESSFUL
[2019/03/27 13:01:58.640062, 0] ../source3/smbd/smb2_sesssetup.c:471(smbd_smb2_auth_generic_return)
  smb2: Failed to update session for vuid=2500254858 - NT_STATUS_UNSUCCESSFUL
[2019/03/27 13:01:58.641668, 0] ../source3/smbd/smbXsrv_session.c:1676(smbXsrv_session_logoff)
  smbXsrv_session_logoff(0x9506dc8a): failed to delete global key '9506DC8A': NT_STATUS_NOT_FOUND
[2019/03/27 13:01:58.641712, 0] ../source3/smbd/smbXsrv_session.c:1192(smbXsrv_session_destructor)
  smbXsrv_session_destructor: smbXsrv_session_logoff() failed: NT_STATUS_NOT_FOUND
[2019/03/27 13:01:58.652245, 0] ../source3/smbd/smbXsrv_session.c:1402(smbXsrv_session_update)
  smbXsrv_session_update: global_id (0x0fecb083) store failed - NT_STATUS_UNSUCCESSFUL

The meassage I get is the following:
The Samba 'panic action' script, /usr/share/samba/panic-action, was called for PID 22658 (/usr/sbin/smbd).

This means there was a problem with the program, such as a segfault.
Below is a backtrace for this process generated with gdb, which shows the state of the program at the time the error occurred. The Samba log files may contain additional information about the problem.

If the problem persists, you are encouraged to first install the samba-dbg package, which contains the debugging symbols for the Samba binaries. Then submit the provided information as a bug report to Ubuntu by visiting this link:
https://launchpad.net/ubuntu/+source/samba/+filebug

[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f55fe00e687 in __GI___waitpid (pid=22659, stat_loc=stat_loc@entry=0x7ffdc4b521b8, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#0 0x00007f55fe00e687 in __GI___waitpid (pid=22659, stat_loc=stat_loc@entry=0x7ffdc4b521b8, options=options@entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:30
#1 0x00007f55fdf79067 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:149
#2 0x00007f55ff6f2b51 in smb_panic_s3 () from /usr/lib/x86_64-linux-gnu/libsmbconf.so.0
#3 0x00007f56017db8cf in smb_panic () from /usr/lib/x86_64-linux-gnu/libsamba-util.so.0
#4 0x000055e931013030 in smbd_accept_connection (ev=0x55e931409a50, fde=<optimized out>, flags=<optimized out>, private_data=<optimized out>) at ../source3/smbd/server.c:1027
#5 0x00007f55fe324ed0 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#6 0x00007f55fe323357 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#7 0x00007f55fe31f7cd in _tevent_loop_once () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#8 0x00007f55fe31f9eb in tevent_common_loop_wait () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#9 0x00007f55fe3232f7 in ?? () from /usr/lib/x86_64-linux-gnu/libtevent.so.0
#10 0x000055e93100e34a in smbd_parent_loop (parent=0x55e93141a2d0, ev_ctx=0x55e931409a50) at ../source3/smbd/server.c:1397
#11 main (argc=<optimized out>, argv=<optimized out>) at ../source3/smbd/server.c:2164 A debugging session is active.

 Inferior 1 [process 22658] will be detached.

Quit anyway? (y or n) [answered Y; input not from terminal]

Versions:

Ubuntu 18.04.2 LTS
Samba 4.7.6
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2011-07-27 (2801 days ago)
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
NmbdLog: messaging_reinit() failed: NT_STATUS_DISK_FULL
Package: samba 2:4.7.6+dfsg~ubuntu-0ubuntu2.6
PackageArchitecture: amd64
ProcVersionSignature: Ubuntu 4.15.0-46.49-generic 4.15.18
SambaServerRegression: Yes
SmbConfIncluded: Yes
Tags: bionic
TestparmExitCode: 0
Uname: Linux 4.15.0-46-generic x86_64
UpgradeStatus: Upgraded to bionic on 2018-08-22 (218 days ago)
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
WindowsFailedConnect: Yes
_MarkForUpload: True
mtime.conffile..etc.cron.daily.samba: 2018-12-13T15:41:00.903970
mtime.conffile..etc.logrotate.d.samba: 2018-10-22T16:19:55.041659

[Christian Ehrhardt  (paelzer)]

@Paride - I think Andreas said we should ask for the .crash file right?

@Marc - could you please attach the full crash file that is create din /var/crash for this?
That would allow to see more than just the backtrace.

To know exactly the versions you have installed atm I think you could also run
 $ apport-collect 1821935

I'm not entirely sure if this also auto-attaches the crash.
If it doesn't you can do so manually after the other info is added.

Setting incomplete waiting for that extra data.

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Marc Vanhoomissen (mvh)]

apport information

[Andreas Hasenack (ahasenack)]

Thanks for the info, I'll look at it.

[Andreas Hasenack (ahasenack)]

The samba logs are full of:
  PANIC (pid 29821): reinit_after_fork() failed
  messaging_reinit() failed: NT_STATUS_DISK_FULL

How is the disk space on that server? Can you show "df -h"?

[Marc Vanhoomissen (mvh)]

df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
udev 457M 0 457M 0% /dev
tmpfs 98M 3,0M 95M 4% /run
/dev/mapper/servamtex2-root 456G 76G 358G 18% /
tmpfs 486M 0 486M 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 486M 0 486M 0% /sys/fs/cgroup
/dev/mapper/servamtex-root 456G 36G 397G 9% /servamtex-old
/dev/sdc 1,8T 1,1T 696G 61% /bu
/dev/sda1 228M 135M 82M 63% /boot
tmpfs 98M 0 98M 0% /run/user/1000

[Andreas Hasenack (ahasenack)]

Can you get a fresh set of these when the panic happens again please?
- df -h
- logs from /var/log/samba/log*
- crash file from /var/crash (you might want to remove any crash files from there right now, to be sure it's a new one when the panic happens again).

[Marc Vanhoomissen (mvh)]

df -h
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
udev 457M 0 457M 0% /dev
tmpfs 98M 39M 59M 40% /run
/dev/mapper/servamtex2-root 456G 76G 358G 18% /
tmpfs 486M 0 486M 0% /dev/shm
tmpfs 5,0M 0 5,0M 0% /run/lock
tmpfs 486M 0 486M 0% /sys/fs/cgroup
/dev/mapper/servamtex-root 456G 36G 397G 9% /servamtex-old
/dev/sdc 1,8T 1,1T 678G 62% /bu
/dev/sda1 228M 135M 82M 63% /boot
tmpfs 98M 0 98M 0% /run/user/1000

[Andreas Hasenack (ahasenack)]

Hm, the logs still say disk full... Could it be that someone is trying to upload a really large file and that is crashing the system? Maybe a backup job?

Do you have anything suspicious in the output of dmesg? Disk issues, controller resets, other disk related errors?

I'll take another look at the crash dump.

[Andreas Hasenack (ahasenack)]

Maybe quotas?e

[Andreas Hasenack (ahasenack)]

Could you set log level = 2 please, and then send the logs again when the panic happens?

[Marc Vanhoomissen (mvh)]

Could you please tell me where I have to set that level?

Thank you.

Marc Vanhoomissen

Le 10-04-19 à 15:32, Andreas Hasenack a écrit :
> Could you set log level = 2 please, and then send the logs again when
> the panic happens?
>

[Marc Vanhoomissen (mvh)]

Since last comment, no problem anymore. An update for samba has been automatically installed in the mean time, as well as a couple of windows 10 updates.