| 2021-01-18 13:52:46 |
Christian Ehrhardt |
bug |
|
|
added bug |
| 2021-01-18 13:52:58 |
Christian Ehrhardt |
summary |
qemu can't access files that are added as rules |
qemu can't access files that are added as rules on hot-add |
|
| 2021-01-18 15:18:05 |
Christian Ehrhardt |
description |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was taking my new code out of the equation - and to do so I was adding the path that is needed to the local overrides. But it still is blocked. So - right now - I'm assuming that my code worked in adding the lines, but the access is still blocked. Therefore let us try to fix this one first and only then I can debug into what might be failing on the new code.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xm
Libvirt/Qemu works as usual - except (known and expected) does not add a rule for "/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2". But for that we have added out local override.
Yet it is denied access.
apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new_<var-snap-lxd-common-lxd>" profile="libvirt-a2caaf89-0682-464d-92ba-5295cb5f5128" name="/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" pid=2889044 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. The rule is generally working
To be clear, on the same system if I just open up a new profile and allow this path to be accessed it works fine. See the working example below.
It must be somewhat that is tied to the way KVM guests get their profile updates/assigned.
root@h-libvirt-orig:~# cat /etc/apparmor.d/test
#include <tunables/global>
profile test flags=(attach_disconnected) {
#include <abstractions/base>
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/usr/bin/md5sum" rmix,
}
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
[6939] aa_change_onexec("test")
[6939] exec md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
0f7fc62d270b69ee1b51453fa614d3e4 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
[6944] aa_change_onexec("test")
[6944] exec md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
md5sum: /var/lib/libvirt/images/testdisk-snap-base.qcow2: Permission denied
## 4. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it. |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was taking my new code out of the equation - and to do so I was adding the path that is needed to the local overrides. But it still is blocked. So - right now - I'm assuming that my code worked in adding the lines, but the access is still blocked. Therefore let us try to fix this one first and only then I can debug into what might be failing on the new code.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt/Qemu works as usual - except (known and expected) does not add a rule for "/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2". But for that we have added out local override.
Yet it is denied access.
apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new_<var-snap-lxd-common-lxd>" profile="libvirt-a2caaf89-0682-464d-92ba-5295cb5f5128" name="/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" pid=2889044 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. The rule is generally working
To be clear, on the same system if I just open up a new profile and allow this path to be accessed it works fine. See the working example below.
It must be somewhat that is tied to the way KVM guests get their profile updates/assigned.
root@h-libvirt-orig:~# cat /etc/apparmor.d/test
#include <tunables/global>
profile test flags=(attach_disconnected) {
#include <abstractions/base>
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/usr/bin/md5sum" rmix,
}
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
[6939] aa_change_onexec("test")
[6939] exec md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
0f7fc62d270b69ee1b51453fa614d3e4 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
[6944] aa_change_onexec("test")
[6944] exec md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
md5sum: /var/lib/libvirt/images/testdisk-snap-base.qcow2: Permission denied
## 4. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it. |
|
| 2021-01-19 07:54:16 |
Christian Ehrhardt |
description |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was taking my new code out of the equation - and to do so I was adding the path that is needed to the local overrides. But it still is blocked. So - right now - I'm assuming that my code worked in adding the lines, but the access is still blocked. Therefore let us try to fix this one first and only then I can debug into what might be failing on the new code.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt/Qemu works as usual - except (known and expected) does not add a rule for "/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2". But for that we have added out local override.
Yet it is denied access.
apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new_<var-snap-lxd-common-lxd>" profile="libvirt-a2caaf89-0682-464d-92ba-5295cb5f5128" name="/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" pid=2889044 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. The rule is generally working
To be clear, on the same system if I just open up a new profile and allow this path to be accessed it works fine. See the working example below.
It must be somewhat that is tied to the way KVM guests get their profile updates/assigned.
root@h-libvirt-orig:~# cat /etc/apparmor.d/test
#include <tunables/global>
profile test flags=(attach_disconnected) {
#include <abstractions/base>
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/usr/bin/md5sum" rmix,
}
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
[6939] aa_change_onexec("test")
[6939] exec md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
0f7fc62d270b69ee1b51453fa614d3e4 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
[6944] aa_change_onexec("test")
[6944] exec md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
md5sum: /var/lib/libvirt/images/testdisk-snap-base.qcow2: Permission denied
## 4. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it. |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was taking my new code out of the equation - and to do so I was adding the path that is needed to the local overrides. But it still is blocked. So - right now - I'm assuming that my code worked in adding the lines, but the access is still blocked. Therefore let us try to fix this one first and only then I can debug into what might be failing on the new code.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt/Qemu works as usual - except (known and expected) does not add a rule for "/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2". But for that we have added out local override.
Yet it is denied access.
apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new_<var-snap-lxd-common-lxd>" profile="libvirt-a2caaf89-0682-464d-92ba-5295cb5f5128" name="/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" pid=2889044 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. The rule is generally working
To be clear, on the same system if I just open up a new profile and allow this path to be accessed it works fine. See the working example below.
It must be somewhat that is tied to the way KVM guests get their profile updates/assigned.
root@h-libvirt-orig:~# cat /etc/apparmor.d/test
#include <tunables/global>
profile test flags=(attach_disconnected) {
#include <abstractions/base>
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/usr/bin/md5sum" rmix,
}
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
[6939] aa_change_onexec("test")
[6939] exec md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
0f7fc62d270b69ee1b51453fa614d3e4 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
[6944] aa_change_onexec("test")
[6944] exec md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
md5sum: /var/lib/libvirt/images/testdisk-snap-base.qcow2: Permission denied
## 4. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it. |
|
| 2021-01-19 08:28:13 |
Christian Ehrhardt |
description |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was taking my new code out of the equation - and to do so I was adding the path that is needed to the local overrides. But it still is blocked. So - right now - I'm assuming that my code worked in adding the lines, but the access is still blocked. Therefore let us try to fix this one first and only then I can debug into what might be failing on the new code.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt/Qemu works as usual - except (known and expected) does not add a rule for "/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2". But for that we have added out local override.
Yet it is denied access.
apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new_<var-snap-lxd-common-lxd>" profile="libvirt-a2caaf89-0682-464d-92ba-5295cb5f5128" name="/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" pid=2889044 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. The rule is generally working
To be clear, on the same system if I just open up a new profile and allow this path to be accessed it works fine. See the working example below.
It must be somewhat that is tied to the way KVM guests get their profile updates/assigned.
root@h-libvirt-orig:~# cat /etc/apparmor.d/test
#include <tunables/global>
profile test flags=(attach_disconnected) {
#include <abstractions/base>
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/usr/bin/md5sum" rmix,
}
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
[6939] aa_change_onexec("test")
[6939] exec md5sum /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
0f7fc62d270b69ee1b51453fa614d3e4 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
root@h-libvirt-orig:~# apparmor_parser -r /etc/apparmor.d/test; aa-exec -v -p test -- md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
[6944] aa_change_onexec("test")
[6944] exec md5sum /var/lib/libvirt/images/testdisk-snap-base.qcow2
md5sum: /var/lib/libvirt/images/testdisk-snap-base.qcow2: Permission denied
## 4. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it. |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was debugging various cases (see below updates) and found that just one
combination fails (also see comment #5)
I'm rather sure now that libvirt adds the path correctly, but even adding the denied path that is needed to the local overrides does not help.
Use a Hirsute Container and enable this PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4393
Then - to underline that the rule should be ok also add it manually as local override.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
$ echo '"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qem
# Check the profile can be loaded and contains these
$ apparmor_parser -r /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877
$ apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877 | grep snap
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach the disk to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt (as usual) now adds rules for the image file.
With the new code from the PPA it does so twice - once for the snapshot and once the base file.
But in any case we allow that access through our override.
You can see that on a little gimmick, since the flattened profile always has those rules - and when libvirt is adding them we will sometimes now see info="same as current profile, skipping".
If the local override isn't in place that does not show up.
So the rules libvirt adds are identical to what we have put into the local override and that should allow the access.
But we get:
[699826.573035] audit: type=1400 audit(1611044559.393:2047): apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new2_<var-snap-lxd-common-lxd>" profile="libvirt-85410987-d91b-487a-8f2c-911f0136c877" name="/var/lib/libvirt/images/testdisk-snap-base.qcow2" pid=111701 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it.
## "special combination"
As explained in later comments it works on bare metal, it works with the old libvirt code, it works if attaching one disk (no two calls to apparmor_parser from libvirt), it works if I do the same manually.
But still the bad case fails - puzzling - please help. |
|
| 2021-01-19 10:09:32 |
Christian Ehrhardt |
summary |
qemu can't access files that are added as rules on hot-add |
qemu can't access files - even with manual overrides (only an issue in Containers) |
|
| 2021-01-19 10:16:30 |
Christian Ehrhardt |
description |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was debugging various cases (see below updates) and found that just one
combination fails (also see comment #5)
I'm rather sure now that libvirt adds the path correctly, but even adding the denied path that is needed to the local overrides does not help.
Use a Hirsute Container and enable this PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4393
Then - to underline that the rule should be ok also add it manually as local override.
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
$ echo '"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qem
# Check the profile can be loaded and contains these
$ apparmor_parser -r /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877
$ apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877 | grep snap
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach the disk to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk
virsh attach-device f-test2 hot-add-test.xml
Libvirt (as usual) now adds rules for the image file.
With the new code from the PPA it does so twice - once for the snapshot and once the base file.
But in any case we allow that access through our override.
You can see that on a little gimmick, since the flattened profile always has those rules - and when libvirt is adding them we will sometimes now see info="same as current profile, skipping".
If the local override isn't in place that does not show up.
So the rules libvirt adds are identical to what we have put into the local override and that should allow the access.
But we get:
[699826.573035] audit: type=1400 audit(1611044559.393:2047): apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new2_<var-snap-lxd-common-lxd>" profile="libvirt-85410987-d91b-487a-8f2c-911f0136c877" name="/var/lib/libvirt/images/testdisk-snap-base.qcow2" pid=111701 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. tracking apparmor_parser
I was using (for debug) a wrapper for apparmro_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong - why else could the access be denied despite the rule that should allow it.
## "special combination"
As explained in later comments it works on bare metal, it works with the old libvirt code, it works if attaching one disk (no two calls to apparmor_parser from libvirt), it works if I do the same manually.
But still the bad case fails - puzzling - please help. |
Hi,
to be clear I consider it quite likely that the error is on my side, but I'd appreciate guidance how to continue resolve. I have rubber ducked my setup with a coworker and talked to jjohansen if the profile would contain an obvious fault (it did not).
So hereby I'm filing a bug hoping to get some help on this case.
## 1. What happened
I was trying to add a feature to libvirt to support chained qcow files.
That means instead of adding just one path libvirt has to process the chain of backing files and add all of those. That works already on guest start, but not on hot-add of devices.
But while I see my code appending the rules I'd expect and issuing the apparmor_parser calls I'd expect it still fails.
I'd love to get the profile at runtime to ensure things really are loaded as expected.
But as discussed on IRC that won't work. So I tested and simplified and wonder for the test below why things fail.
## 2. how to reproduce
I was debugging various cases (see below updates) and found that just one
combination fails (also see comment #5).
I'm rather sure now that libvirt adds the path correctly, but even adding the denied path that is needed to the local overrides does not help.
Use a Hirsute Container and enable this PPA:
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4393
# First get a KVM capable container (the issue only triggers in containers)
$ lxc profile create kvm
$ cat > kvm_profile.yaml << EOF
config:
boot.autostart: "true"
linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,kvm
security.nesting: "true"
security.privileged: "true"
description: ""
devices:
eth0:
mtu: "9000"
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
kvm:
path: /dev/kvm
type: unix-char
mem:
path: /dev/mem
type: unix-char
tun:
path: /dev/net/tun
type: unix-char
name: kvm
EOF
$ lxc profile edit kvm < kvm_profile.yaml
$ lxc launch ubuntu-daily:h h-libvirt-new --profile default --profile kvm
All following steps are in the container
$ sudo add-apt-repository ppa:ci-train-ppa-service/4393
$ apt update
$ apt upgrade -y
$ apt install uvtool-libvirt
Then - to underline that the rules should be ok - also add it manually as local override (therefore we don't need to theorize or check if the rule was indeed added by libvirt - but since it works on bare metal we know it was added).
# Allow the access that libvirt does not yet allow
$ echo '"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,' > /etc/apparmor.d/local/abstractions/libvirt-qemu
$ echo '"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,' >> /etc/apparmor.d/local/abstractions/libvirt-qemu
# Check the profile can be loaded and contains these
$ apparmor_parser -r /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877
$ apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-85410987-d91b-487a-8f2c-911f0136c877 | grep snap
"/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2" rwk,
"/var/lib/libvirt/images/testdisk-snap-base.qcow2" rwk,
# prep the disk chain to be hot-added
qemu-img create -f qcow2 /var/lib/libvirt/images/testdisk-snap-base.qcow2 100M
qemu-img create -f qcow2 -b /var/lib/libvirt/images/testdisk-snap-base.qcow2 /var/lib/libvirt/images/testdisk-snap-snapshot.qcow2
cat > hot-add-test.xml << EOF
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-snapshot.qcow2' index='1'/>
<backingStore type='file' index='2'>
<format type='qcow2'/>
<source file='/var/lib/libvirt/images/testdisk-snap-base.qcow2'/>
<backingStore/>
</backingStore>
<target dev='vdc' bus='virtio'/>
</disk>
EOF
# prep a guest to attach the disk to
uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=focal
uvt-kvm create --host-passthrough --password=ubuntu f-test2 release=focal arch=amd64 label=daily
We have together checked the "effective" profile that is loaded. JJohanssen didn't see an obvious flaw in it. But for reference here:
root@h-libvirt-orig:~# apparmor_parser -p /etc/apparmor.d/libvirt/libvirt-e033b910-be06-4975-826f-b6fba368d928 | pastebinit
https://paste.ubuntu.com/p/yxdmMT6638/
# attach the disk to trigger the issue
virsh attach-device f-test2 hot-add-test.xml
Libvirt (as usual) now adds rules for the image file (seen in dmesg).
With the new code from the PPA it does so twice - once for the snapshot and once the base file.
But in any case we allow that access through our override.
You can see that at a little gimmick, since the flattened profile always has those rules - and when libvirt is adding them we will sometimes now see info="same as current profile, skipping".
If the local override isn't in place that does not show up.
So the rules libvirt adds are identical to what we have put into the local override and that should allow the access.
But we get:
[699826.573035] audit: type=1400 audit(1611044559.393:2047): apparmor="DENIED" operation="open" namespace="root//lxd-h-libvirt-new2_<var-snap-lxd-common-lxd>" profile="libvirt-85410987-d91b-487a-8f2c-911f0136c877" name="/var/lib/libvirt/images/testdisk-snap-base.qcow2" pid=111701 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=64055
## 3. tracking apparmor_parser
I was using (for debug) a wrapper for apparmor_parser but all really looks as expected
$ cat /usr/sbin/apparmor_parser.wrap
#!/bin/bash
echo "ARGS $@" | /usr/bin/systemd-cat
echo "Content of ${2}:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}" | /usr/bin/systemd-cat
/usr/bin/cat "${2}" | /usr/bin/systemd-cat
echo "Content of ${2}.files:" | /usr/bin/systemd-cat
/usr/bin/ls -laF "${2}.files" | /usr/bin/systemd-cat
/usr/bin/cat "${2}.files" | /usr/bin/systemd-cat
/usr/sbin/apparmor_parser.orig "$@" | /usr/bin/systemd-cat
It is called with "-r filepath" and the content of that filepath contain the aforementioned rules. So what else might be wrong? Why else could the access be denied despite the rule that should allow it being present?
Also do mind that it blocks only the secondary access - which means it has read the snapshot file and determined that it also needs to open the base file
## "special combination"
As explained in later comments it works on bare metal, it works with the old libvirt code, it works if attaching just one disk (no two calls to apparmor_parser from libvirt), it works if I do the same manually.
But still the bad case fails - puzzling - please help. |
|
