docker snap suddenly installed by itself

[A Z]

Have an Ubuntu 18.04 VM with docker.io installed from repo, it suddenly ran out of diskspace after the docker snap installed all by itself at 01:00 Amsterdam time.
Both snap docker and docker from repo were fighting over /run/containerd/containerd.sock:

/var/log/syslog shows many of below lines being added:

docker.dockerd[5786]: time="2019-11-15T01:54:57.345655308Z" level=error msg="failed to get event" error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure,
latest connection error: connection error: desc = \"transport: Error while dialing dial unix /run/containerd/containerd.sock: connect: permission denied\"" module=libcontainerd namespace=moby

On checking the file:

sudo fuser /run/containerd/containerd.sock
1052
ps aux | grep 1052
root 1026 219 1.1 2010520 23976 ? Ssl 09:07 33:18 dockerd -G docker --exec-root=/var/snap/docker/418/run/docker --data-root=/var/snap/docker/common/var-lib-docker --pidfile=/var/snap/docker/418/run/docker.pid --config-file=/var/snap/docker/418/config/daemon.json
root 1052 0.2 0.2 2650780 5540 ? Ssl 09:07 0:02 /usr/bin/containerd

Similar to what is described here: https://askubuntu.com/questions/1188908/what-does-this-docker-log-entry-mean
I've removed the snap and snapd for now and cleared the syslog.

What's going on?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[lubobill1990 (lubobill1990)]

I regard this is not just a bug, but a serious security hazard. Why snap can automatically install docker from outside control?

[A Z (azaagman)]

Adding in workaround:

Remove either the docker snap or the docker.io package:

sudo snap remove docker
OR
sudo apt purge docker.io

Truncate the syslog to get your diskspace back:

sudo truncate -s 0 /var/log/syslog

[Daniel Skowroński (danielskowronski)]

I second that this is security hazard. This may be related to unattended-upgrades package which could interact with snap, but I don't have machine which was affected by this (but I'm sure it was there).

Can anyone with the issue active backup syslog before truncating it so it can be investigated?

[Oliver Grawert (ogra)]

unattended-updates do not deal with snap packages (effectively snaps care for their upgrades themselves, there is no external entity like unattended-upgrades needed)...

also snap packages do not "just install themselves", something must have actively triggered that ...

along with syslog (or rather journald output nowadays) the output of "snap changes" would be interesting as well ...

[Guy Wyant (gwyant)]

Same symptoms as above for us. Totally unattended server and we were hit with an outage immediately after this change happened.

First line is the relevant snap changes entry of where the problems began. Then we removed the docker snap today at 10:39.

redacted@redacted:/# snap changes
ID Status Spawn Ready Summary
33 Done yesterday at 19:38 EST yesterday at 19:39 EST Auto-refresh snap "docker"
35 Done today at 10:39 EST today at 10:39 EST Remove "docker" snap

[Oliver Grawert (ogra)]

sadly snapd wipes the logs after a while so your snap changes start only at change 33 and this is only a refresh of the already installed docker snap ...

it would be interesting to know why/how it got installed in the first place, perhaps someone from the snapd team has a hint here ...

do you have any docker management tools installed that might have pulled in the docker snap ?

[mike kershaw (dragorn)]

Also hit here on a near-vanilla 18.04 install; snap docker auto-updated and fell down, and stayed down until manually reloaded.

$ snap changes
ID Status Spawn Ready Summary
10 Done yesterday at 23:06 EST yesterday at 23:06 EST Auto-refresh snap "docker"

[Guy Wyant (gwyant)]

Well, regardless of whether snap docker was already installed prior to change 33, it was at that auto-refresh when the issue actually began, not before.

No docker tools we have would have pulled in docker snap automatically, and we had not changed a thing on the server for days.

[Don Gi Min (dongimin2)]

I have this exact issue, and I have a ~200GB syslog file because of it. I'm not experienced enough to do forensics on this, if anyone has some idea where to look I'll help.

[Tianon Gravi (tianon)]

At least for the "Docker fails to run containers and fill syslog" half of this, I've reproduced in https://forum.snapcraft.io/t/docker-snap-18-09-9-error/14171/5 and have a fix committed waiting for Launchpad to build so I can rush it through the channels ASAP.

[Tianon Gravi (tianon)]

My fix finished building and uploading into the edge channel, so if folks could test it from either edge or beta (I'd suggest beta, but your choice), that would be helpful.

[DataBitz (databitz)]

Thanks, tested from beta channel and it seems to have fixed the problem for me

[Linrav (linrav)]

Hi @Tianon. Confirmed identical symptoms and issue with snap clashing over ubuntu's containerd! Refreshed your snap from edge channel and confirmed it fixes the issue. Thank you so much! Saved the day :)

[Ishan (ishan-bn14)]

I had docker with apt and snap package, suddenly today i was seeing my cpu full and disk space full with logs .
and had a permission error in the logs with contaired.

[Tianon Gravi (tianon)]

Thanks for testing and confirming! Fix is now released to the stable channel.

[John Lenton (chipaca)]

I'm setting as invalid for snapd, as there is nothing snapd is doing wrong here AFAICT.

[John Lenton (chipaca)]

(if you disagree please set back from Invalid as otherwise it won't get picked up by our triage).

[Björn Svensson (bsv116)]

I would like you to open the case for Snap again. On my machine, it continues to auto install other packages as well. I just got chromium today. Similar to the case with docker, the machine starts to behave strange when there are dual installations of applications. Similar to the docker fix, I can remove it in Snap and the machine starts to behave again. It seems like Snap makes unrequested updates of packages that was not installed via Snap. The problem seems to be with Snap rather than Docker.

[Zygmunt Krynicki (zyga)]

Chromium deb is automatically migrating to the snap. This is a specific case and is not related to docker.

[olivier (beyler-olivier)]

I please note that the snap may not be the root cause. I've got the same pb without have installed snap.
see my issue on GitHub https://github.com/moby/moby/issues/40390