[UBUNTU 24.04] OS guest boot issues on 9p filesystem
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Ubuntu on IBM z Systems |
Fix Released
|
Critical
|
Skipper Bug Screeners | ||
| qemu (Ubuntu) |
Fix Released
|
Critical
|
Sergio Durigan Junior | ||
| Noble |
Fix Released
|
Undecided
|
Sergio Durigan Junior | ||
| Oracular |
Fix Released
|
Undecided
|
Sergio Durigan Junior | ||
| Plucky |
Fix Released
|
Critical
|
Sergio Durigan Junior | ||
Bug Description
[ Impact ]
Users of the 9p filesystem may encounter problems when guests try to access host files, which can lead to e.g. guest boot issues.
[ Test plan ]
Using the script kindly provided by the reporter, we can set up our test environment as such (this requires an s390x machine):
$ lxc profile create lp2091099
$ lxc profile edit lp2091099 << _EOF_
name: kvm-nodisk
description: ""
config:
boot.autostart: "true"
linux.
security.nesting: "true"
security.
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
kvm:
path: /dev/kvm
type: unix-char
mem:
path: /dev/mem
type: unix-char
tun:
path: /dev/net/tun
type: unix-char
_EOF_
$ lxc launch ubuntu-daily:noble n-9ptest --profile default --profile lp2091099
$ lxc shell n-9ptest
# apt update
# apt install -y qemu-system-s390x debootstrap
# cat > t.sh << _EOF_
#!/bin/bash
# Cleanup target dir
[ -d ./target ] && rm -rf target
mkdir target
# Add configuration updates
mkdir -p ./target/
echo 9p >> ./target/
echo 9pnet_virtio >> ./target/
# Add the test script
cat > ./target/test_init << EOF
#!/bin/bash
echo "Test for unix domain sockets"
nc -Ul /socket &
sleep 1
echo "Sockets work" | nc -UN /socket || echo "Sockets fail"
echo o > /proc/sysrq-trigger
sleep 999
EOF
chmod 700 ./target/test_init
# Create an Ubuntu 23.10 around it
echo "Creating Ubuntu target OS"
debootstrap --variant=minbase\
noble ./target > /dev/null || exit 1
# Run the test in 9p forwarded filesystem
echo "Running OS in qemu"
qemu-system-s390x \
-m 8192 \
-smp 4 \
-nodefaults -nographic -no-reboot -no-user-config \
-kernel ./target/
-initrd ./target/
-append 'root=fsRoot rw rootfstype=9p rootflags=
-fsdev local,security_
-device virtio-
-device virtio-serial-ccw -device sclpconsole,
-chardev stdio,id=
_EOF_
# sh ./t.sh
On a problematic system, you should see something similar to the following output:
Creating Ubuntu target OS
Running OS in qemu
KASLR disabled: CPU has no PRNG
KASLR disabled: CPU has no PRNG
Test for unix domain sockets
qemu-system-s390x: 9p: broken or compromised client detected; attempt to open special file (i.e. neither regular file, nor directory)
nc: No such device or address
nc: /socket: No such file or directory
Sockets fail
[ Where problems could occur ]
This failure is caused by a regression introduced by a fix to a CVE. As such, the possibility of introducing a security regression needs to be considered. The CVE fix was about preventing guests to open files on the host when using 9p, but it ended up being too restrictive and impacting use cases when there needs to be an access.
The patch is not too complex and is specifically aimed at allowing open(2) with O_PATH, which is an operation that is as secure as possible without actually opening the file. According to the manpage:
O_PATH (since Linux 2.6.39)
Obtain a file descriptor that can be used for two purposes: to indicate a location in the filesystem tree and to perform operations that act purely at the file descriptor level. The file itself is not opened, and other file operations (e.g., read(2), write(2), fchmod(2), fchown(2), fgetxattr(2), ioctl(2), mmap(2)) fail with the error EBADF.
[ Original Description ]
== Reported by <email address hidden> - 2024-12-04 ==
---Problem Description---
[UBUNTU 24.04] OS guest boot issues on 9p filesystem
---uname output---
Linux winlnxnw 6.8.0-45-generic #45-Ubuntu SMP Fri Aug 30 11:09:37 UTC 2024 s390x s390x s390x GNU/Linux
Contact Information = <email address hidden>
Machine Type = 3931-7G4
---Steps to Reproduce---
Run this script (fails with a qemu error message)
#!/bin/bash
# Cleanup target dir
[ -d ./target ] && rm -rf target
mkdir target
# Add configuration updates
mkdir -p ./target/
echo 9p >> ./target/
echo 9pnet_virtio >> ./target/
# Add the test script
cat > ./target/test_init << EOF
#!/bin/bash
echo "Test for unix domain sockets"
nc -Ul /socket &
sleep 1
echo "Sockets work" | nc -UN /socket || echo "Sockets fail"
echo o > /proc/sysrq-trigger
sleep 999
EOF
chmod 700 ./target/test_init
# Create an Ubuntu 23.10 around it
echo "Creating Ubuntu target OS"
debootstrap --variant=minbase\
noble ./target > /dev/null || exit 1
# Run the test in 9p forwarded filesystem
echo "Running OS in qemu"
qemu-system-s390x \
-m 8192 \
-smp 4 \
-nodefaults -nographic -no-reboot -no-user-config \
-kernel ./target/
-initrd ./target/
-append 'root=fsRoot rw rootfstype=9p rootflags=
-fsdev local,security_
-device virtio-
-device virtio-serial-ccw -device sclpconsole,
-chardev stdio,id=
---Debugger---
A debugger is not configured
Userspace tool obtained from project website: na
Userspace rpm: qemu-(current).deb
Userspace tool common name: qemu
The userspace tool has the following bit modes: both
*Additional Instructions for <email address hidden>:
-Post a private note with access information to the machine that the bug is occuring on.
-Attach ltrace and strace of userspace application.
CVE References
| tags: | added: architecture-s39064 bugnameltc-210802 severity-critical targetmilestone-inin--- |
| Changed in ubuntu: | |
| assignee: | nobody → Skipper Bug Screeners (skipper-screen-team) |
| affects: | ubuntu → linux (Ubuntu) |
| affects: | linux (Ubuntu) → qemu (Ubuntu) |
| Changed in ubuntu-z-systems: | |
| assignee: | nobody → Skipper Bug Screeners (skipper-screen-team) |
| Changed in qemu (Ubuntu): | |
| importance: | Undecided → Critical |
| Changed in qemu (Ubuntu): | |
| assignee: | Skipper Bug Screeners (skipper-screen-team) → Sergio Durigan Junior (sergiodj) |
| tags: | added: server-todo |
| Changed in ubuntu-z-systems: | |
| importance: | Undecided → Critical |
| status: | New → Triaged |
| description: | updated |
| description: | updated |
| Changed in qemu (Ubuntu Plucky): | |
| status: | New → Fix Committed |
| Changed in qemu (Ubuntu Oracular): | |
| assignee: | nobody → Sergio Durigan Junior (sergiodj) |
| Changed in qemu (Ubuntu Noble): | |
| assignee: | nobody → Sergio Durigan Junior (sergiodj) |
| Changed in ubuntu-z-systems: | |
| status: | Triaged → In Progress |
| Changed in ubuntu-z-systems: | |
| status: | In Progress → Fix Committed |
| Changed in ubuntu-z-systems: | |
| status: | Fix Committed → Fix Released |
| tags: |
added: targetmilestone-inin2404 removed: targetmilestone-inin--- |

------- Comment From <email address hidden> 2024-12-05 07:22 EDT------- 2ubuntu6. 21)
This bug was already detected in Jammy and was fixed in Ubuntu 22.04 (package qemu - 1:6.2+dfsg-
==> see bugzilla / LP item 'Bug 206380 - LP2065579 : [UBUNTU 22.04] OS guest boot issues on 9p filesystem'
Now, the problem seems to be back with Noble.
------- Comment From <email address hidden> 2024-12-05 07:25 EDT-------
From the original Jammy bug:
This Bug is the result of the fix to:
CVE-2023-2861: Prohibit opening any special file directly on host
I also opened a Bug in the qemu bugtracker /gitlab. com/qemu- project/ qemu/-/ issues/ 2337
https:/
The containers fail because syslog cannot open its unix domain socket on the filesystem.
We tracked the change that provokes this error to a CVE change in qemu that forbids opening of special files to
prevent exposing data from the host. Special files should be handled by the guest os.
Unix domain socket files are also special files, and they are handled by the guest OS in their entirety, and the 9p server in qemu assigns them individual inodes so they are safe to open. But they must be opened so their fd can be passed to the appropriate connect() or bind() function so the OS can use them.
Socket files don't have a traditional read or write functionality, they are mere representatives for a local address.
There is no convention for where domain socket files should go, so there is no easy fix by just creating a tmpfs somewhere.
We also see other workloads and services failing for not being able to open their local socket files.
The analysis of CVE-2023-2861 in detail reveals
- opening of device files through the 9p server directly grants access to read/write functions of those device files. Also device files can be created in-place anywhere.
- opening of FIFOs is somewhat unsafe as long as there are possible collisions that could expose host data using read/write.
- opening of sockets is safe because the 9p server protects the revealed inode and provides no way to connect the file to a socket.
The qemu team has made a change, but that only made things different, not better.