[2.51.4] stale device cgroups were not removed when a snap got installed in devmode

[Dmitrii Shcherbakov]

I am getting EPERM from the kernel when I try to open a character file when a snap runs in devmode. I would expect devmode to work differently and avoid those rules from being enforced.

$ snap list | grep snapd
snapd 2.51.4 12883 latest/stable canonical* snapd

$ snap list | grep microstack
microstack ussuri 233 latest/beta canonical* devmode

$ sudo snap run --shell microstack.virsh
root@node-hagecius:/home/ubuntu/src/microstack# cat /dev/kvm
cat: /dev/kvm: Operation not permitted

# The rule for /dev/kvm is not in the devices.list
$ sudo grep 232 /sys/fs/cgroup/devices/snap.microstack.virsh/devices.list ; echo $?
1

$ stat /dev/kvm
  File: /dev/kvm
  Size: 0 Blocks: 0 IO Block: 4096 character special file
Device: 5h/5d Inode: 585 Links: 1 Device type: a,e8
Access: (0660/crw-rw----) Uid: ( 0/ root) Gid: ( 115/ kvm)
Access: 2021-09-01 14:21:06.835559208 +0000
Modify: 2021-09-01 14:21:06.835559208 +0000
Change: 2021-09-01 14:21:06.835559208 +0000
 Birth: -

See the bpftrace below:

root@node-hagecius:/home/ubuntu# cat /var/snap/bpftrace/common/trace-cgperm.bt

kfunc:do_sys_openat2 /comm == "cat"/ {
    @pid[tid] = pid;
}

kretfunc:devcgroup_check_permission /@pid[tid]/ {
    printf("cgroup permission check (devcgroup_check_permission) for: major: %d (0x%x), minor: %d (0x%x)\n", args->major, args->major, args->minor, args->minor);
    printf("kstack:\n%s\n", kstack());
    printf("ustack:\n%s\n", ustack());
    printf("retval %d\n", retval);
}

root@node-hagecius:/home/ubuntu# bpftrace /var/snap/bpftrace/common/trace-cgperm.bt
Attaching 2 probes...
cgroup permission check (devcgroup_check_permission) for: major: 10 (0xa), minor: 232 (0xe8)
kstack:

        bpf_prog_3825bd0af8e8dd5f_devcgroup_check+202
        bpf_prog_3825bd0af8e8dd5f_devcgroup_check+202
        bpf_trampoline_6442502014_0+85
        devcgroup_check_permission+5
        inode_permission+20
        may_open+105
        do_open+132
        path_openat+266
        do_filp_open+140
        do_sys_openat2+155
        __x64_sys_openat+86
        do_syscall_64+56
        entry_SYSCALL_64_after_hwframe+68

ustack:

        0x7f243458aeab
        0x2f3d4c4c45485300

retval -1

[Dmitrii Shcherbakov (dmitriis)]

I was testing confined MicroStack (without devmode) on this machine previously.

Looking at this again, I can see that cgroup entries got left untouched even after the snap (in devmode) was purged.

$ sudo snap remove microstack --purge
microstack removed

$ find /sys/fs/cgroup/devices/ -name 'snap.microstack.*'
/sys/fs/cgroup/devices/snap.microstack.keystone-uwsgi
/sys/fs/cgroup/devices/snap.microstack.rabbitmq-server
/sys/fs/cgroup/devices/snap.microstack.openstack
/sys/fs/cgroup/devices/snap.microstack.cinder-scheduler
/sys/fs/cgroup/devices/snap.microstack.cinder-volume
/sys/fs/cgroup/devices/snap.microstack.mysqld
/sys/fs/cgroup/devices/snap.microstack.nova-api-metadata
/sys/fs/cgroup/devices/snap.microstack.ovn-controller
/sys/fs/cgroup/devices/snap.microstack.nova-spicehtml5proxy
/sys/fs/cgroup/devices/snap.microstack.microstack
/sys/fs/cgroup/devices/snap.microstack.libvirtd
/sys/fs/cgroup/devices/snap.microstack.init
/sys/fs/cgroup/devices/snap.microstack.neutron-api
/sys/fs/cgroup/devices/snap.microstack.neutron-ovn-metadata-agent
/sys/fs/cgroup/devices/snap.microstack.ovn-ovsdb-server-nb
/sys/fs/cgroup/devices/snap.microstack.ovsdb-server
/sys/fs/cgroup/devices/snap.microstack.iscsid
/sys/fs/cgroup/devices/snap.microstack.target
/sys/fs/cgroup/devices/snap.microstack.glance-api
/sys/fs/cgroup/devices/snap.microstack.nova-conductor
/sys/fs/cgroup/devices/snap.microstack.virtlogd
/sys/fs/cgroup/devices/snap.microstack.cinder-uwsgi
/sys/fs/cgroup/devices/snap.microstack.cluster-uwsgi
/sys/fs/cgroup/devices/snap.microstack.nginx
/sys/fs/cgroup/devices/snap.microstack.external-bridge
/sys/fs/cgroup/devices/snap.microstack.launch
/sys/fs/cgroup/devices/snap.microstack.ovn-ovsdb-server-sb
/sys/fs/cgroup/devices/snap.microstack.nova-compute
/sys/fs/cgroup/devices/snap.microstack.placement-uwsgi
/sys/fs/cgroup/devices/snap.microstack.virsh
/sys/fs/cgroup/devices/snap.microstack.nova-scheduler
/sys/fs/cgroup/devices/snap.microstack.horizon-uwsgi
/sys/fs/cgroup/devices/snap.microstack.ovs-vswitchd
/sys/fs/cgroup/devices/snap.microstack.ovn-northd
/sys/fs/cgroup/devices/snap.microstack.memcached
/sys/fs/cgroup/devices/snap.microstack.hook.remove
/sys/fs/cgroup/devices/snap.microstack.nova-api

[Dmitrii Shcherbakov (dmitriis)]

I rebooted the host just to get rid of any stale state.

$ ls -1 /sys/fs/cgroup/devices/snap.microstack*
ls: cannot access '/sys/fs/cgroup/devices/snap.microstack*': No such file or directory

After installing the snap in devmode

$ sudo snap install ./microstack_ussuri_amd64.snap --devmode

I do not see any device cgroup entries present:

$ ls -1 /sys/fs/cgroup/devices/snap.microstack*
ls: cannot access '/sys/fs/cgroup/devices/snap.microstack*': No such file or directory

Looks like the residual device cgroup state wasn't cleaned up originally but no new state is created in devmode.

[Ian Johnson (anonymouse67)]

This bug is a duplicate of 1803210

[Dmitrii Shcherbakov (dmitriis)]

Thanks for the pointer, Ian!