OSError: [Errno 30] Read-only file system: '/var/crash/...

[Ralf Hildebrandt]

Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: Traceback (most recent call last):
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/bin/pyzor", line 408, in <module>
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: main()
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/bin/pyzor", line 152, in main
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: if not dispatch(client, servers, config):
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/bin/pyzor", line 239, in check
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: send_digest(digested, mock_runner, servers)
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/bin/pyzor", line 262, in send_digest
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: _send_digest(runner, servers[0], digested)
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/bin/pyzor", line 253, in _send_digest
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: runner.run(server, (digested, server))
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/lib/python3/dist-packages/pyzor/client.py", line 258, in run
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: response = self.routine(*args, **kwargs)
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/lib/python3/dist-packages/pyzor/client.py", line 120, in _mock_check
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: msg = (b"Code: %s\nDiag: OK\nPV: %s\nThread: 1024\nCount: 0\n"
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: TypeError: %b requires a bytes-like object, or an object that implements __bytes__, not 'int'
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: Error in sys.excepthook:
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: Traceback (most recent call last):
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 153, in apport_excepthook
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: with os.fdopen(os.open(pr_filename,
Dec 6 12:02:32 mail-cbf-int pyzor[2966924]: OSError: [Errno 30] Read-only file system: '/var/crash/_usr_bin_pyzor.122.crash'

1) due to an encoding error, pyzor is throwing an exception.
2) /usr/lib/python3/dist-packages/apport_python_hook.py is the exception handler invoked (belonging to the python3-apport package)
3) This exception handler tries to write to /var/crash/_usr_bin_pyzor.122.crash, which fails

I checked the permissions:

# ls -l /var/crash
total 0

# ls -ld /var/crash
drwxrwxrwt 2 root root 4096 Dec 8 16:45 /var/crash

# ls -ld /var/
drwxr-xr-x 18 root root 4096 Jun 23 2020 /var/

# ls -ld /
drwxr-xr-x 24 root root 4096 Nov 30 06:19 /

/var/crash is sticky, so why can't it write?

# cat /proc/mounts |fgrep crash
#

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: python3-apport 2.20.11-0ubuntu27.21
ProcVersionSignature: Ubuntu 5.11.0-36.40~20.04.1-generic 5.11.22
Uname: Linux 5.11.0-36-generic x86_64
ApportLog:

ApportVersion: 2.20.11-0ubuntu27.21
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Dec 9 09:39:40 2021
PackageArchitecture: all
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: apport
UpgradeStatus: Upgraded to focal on 2021-04-07 (245 days ago)

[Benjamin Drung (bdrung)]

The error says "Read-only file system". Is / (or /var) mounted read-only? Did you had a file-system failure where the kernel remounted the file system in read-only mode?

[Ralf Hildebrandt (ralf-hildebrandt)]

No, /var is not mounted read-only. And no, no file-system failure.

[Benjamin Drung (bdrung)]

I still fail to understand what went wrong there. /var/crash has correct permission to write to. In case of permission errors, the error should say PermissionError (errno 13 IIRC), but the failure message says "Read-only file system".

Can you paste the content of /proc/mounts?

What happens if you try to create a file in /var/crash? E.g. run

touch /var/crash/testcase.crash

I would expect that to work.

[Ralf Hildebrandt (ralf-hildebrandt)]

* Benjamin Drung <email address hidden>:
> I still fail to understand what went wrong there. /var/crash has correct
> permission to write to. In case of permission errors, the error should
> say PermissionError (errno 13 IIRC), but the failure message says "Read-
> only file system".
>
> Can you paste the content of /proc/mounts?

# cat /proc/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs
rw,nosuid,noexec,relatime,size=32868660k,nr_inodes=8217165,mode=755,inode64
0 0
devpts /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs
rw,nosuid,nodev,noexec,relatime,size=6577308k,mode=755,inode64 0 0
/dev/sda1 / ext4 rw,relatime,errors=remount-ro,stripe=64 0 0
securityfs /sys/kernel/security securityfs
rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0
tmpfs /run/lock tmpfs
rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755,inode64 0 0
cgroup2 /sys/fs/cgroup/unified cgroup2
rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0
cgroup /sys/fs/cgroup/systemd cgroup
rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
cgroup /sys/fs/cgroup/cpuset cgroup
rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup
rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/freezer cgroup
rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/perf_event cgroup
rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/memory cgroup
rw,nosuid,nodev,noexec,relatime,memory 0 0
cgroup /sys/fs/cgroup/devices cgroup
rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/blkio cgroup
rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma
0 0
cgroup /sys/fs/cgroup/hugetlb cgroup
rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/misc cgroup rw,nosuid,nodev,noexec,relatime,misc
0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids
0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs
rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=22944
0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/debug/tracing tracefs
rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl
rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc
rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/user/0 tmpfs
rw,nosuid,nodev,relatime,size=6577304k,mode=700,inode64 0 0

> What happens if you try to create a file in /var/crash?...

[Benjamin Drung (bdrung)]

/var/crash is on following mount point:

/dev/sda1 / ext4 rw,relatime,errors=remount-ro,stripe=64 0 0

which is read/write. So that should be not the issue.

I just had an idea: Could this issue be caused by a security confinement? In which environment do you run pyzor? Do you use a systemd service? If yes, what security features of systemd do you use?

[Launchpad Janitor (janitor)]

[Expired for apport (Ubuntu) because there has been no activity for 60 days.]