Random errors from deveice mapper driver

[Ilya Dmitrichenko]

Docker behaves very flaky with the devicemapper driver, below is an example of me trying to run the same command in the timespan of under a minute.

ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
2015/01/16 12:22:55 Error response from daemon: open /dev/mapper/docker-8:5-260340-1d147f83d0034a170caaa2db3ddc55df038d61d696ec43860cb6e86bbd1a9297-init: permission denied
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
2015/01/16 12:22:58 Error response from daemon: Cannot start container b7006f8dbbc46330338ce8dd407e0076a3ee16eb45671b447bb783997811be84: Error getting container b7006f8dbbc46330338ce8dd407e0076a3ee16eb45671b447bb783997811be84 from driver devicemapper: open /dev/mapper/docker-8:5-260340-b7006f8dbbc46330338ce8dd407e0076a3ee16eb45671b447bb783997811be84: permission denied
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
2015/01/16 12:22:59 Error response from daemon: open /dev/mapper/docker-8:5-260340-8539e52fdcf48f337888955e6aad533b31b9a4daf008e76fec3d5d1950901b2a-init: permission denied
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
2015/01/16 12:23:30 Error response from daemon: Cannot start container d8dc4386cd0accc815551eb4a8e3bbfffa44d221b0fc9fc1017cd417b82d43f4: Error getting container d8dc4386cd0accc815551eb4a8e3bbfffa44d221b0fc9fc1017cd417b82d43f4 from driver devicemapper: open /dev/mapper/docker-8:5-260340-d8dc4386cd0accc815551eb4a8e3bbfffa44d221b0fc9fc1017cd417b82d43f4: permission denied
ubuntu@localhost:~$ docker run -ti errordeveloper/curl:latest curl
curl: try 'curl --help' for more information
ubuntu@localhost:~$

[Ilya Dmitrichenko (errordeveloper)]

A good start would to upgrade to latest docker 1.4.

[Jamie Strandboge (jdstrand)]

Can you attach the output of:
$ sudo journalctl -k --no-pager|grep DEN

[Ilya Dmitrichenko (errordeveloper)]

Part Tag Installed Available Fingerprint Active
ubuntu-core edge 142 - 18d8361edb919f *
docker edge 1.3.3.001 - 60b98945e5fc1d *

[Ilya Dmitrichenko (errordeveloper)]

Jan 16 12:27:24 localhost.localdomain kernel: audit: type=1400 audit(1421411244.242:10): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-1" pid=560 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:40 localhost.localdomain kernel: audit: type=1400 audit(1421413780.318:11): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-2" pid=605 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:43 localhost.localdomain kernel: audit: type=1400 audit(1421413783.246:12): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-3" pid=571 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:46 localhost.localdomain kernel: audit: type=1400 audit(1421413786.122:13): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-4" pid=585 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:47 localhost.localdomain kernel: audit: type=1400 audit(1421413787.786:14): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-5" pid=571 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:49 localhost.localdomain kernel: audit: type=1400 audit(1421413789.390:15): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-6" pid=571 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:50 localhost.localdomain kernel: audit: type=1400 audit(1421413790.058:16): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-7" pid=560 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 16 13:09:52 localhost.localdomain kernel: audit: type=1400 audit(1421413792.518:17): apparmor="DENIED" operation="open" profile="docker_docker_1.3.3.001" name="/dev/dm-8" pid=571 comm="docker" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Ilya Dmitrichenko (errordeveloper)]

> grep dev /var/lib/apparmor/profiles/profile_docker_docker_1.3.3.001
  /dev/mapper/control rw,
  /dev/mapper/docker* rw,
  /dev/loop* r,
  /dev/loop[0-9]* w,
  /dev/ptmx rw,

I added ` /dev/dm-* rw,` and it seem to be working now.

[Jamie Strandboge (jdstrand)]

Pierre, in you next docker upload, can you add:
   /dev/dm-* rw,

to the apparmor profile? Thanks!

[Jamie Strandboge (jdstrand)]

After talking with Ilya, can you also add:
  /dev/net/ r,
  /dev/snd/ r,
  /dev/ r,
  /dev/block/ r,
  /dev/bsg/ r,
  /dev/char/ r,
  /dev/cpu/ r,
  /dev/disk/ r,
  /dev/disk/by-id/ r,
  /dev/disk/by-label/ r,
  /dev/disk/by-partlabel/ r,
  /dev/disk/by-partuuid/ r,
  /dev/disk/by-path/ r,
  /dev/disk/by-uuid/ r,
  /dev/hugepages/ r,
  /dev/input/ r,
  /dev/input/by-path/ r,

We will probably need to add more accesses since the about only allows reads on directories.

[Ilya Dmitrichenko (errordeveloper)]

Few more I ended-up having to add are:

   /proc r,
   /dev/mapper/ r,
   /dev/mqueue/ r,

I can see there is `@{PROC}/** r`, but I was still seeing denials mentioning `/proc` without leading slash.

[Kick In (kick-d)]

Sorry, I missed this. Doing it right now.

[Ilya Dmitrichenko (errordeveloper)]

Pierre, you forgot to add this:

 /dev/mapper/ r,
 /dev/mqueue/ r,

And, could you verify whether `@{PROC}/** r` doesn't imply `/proc r`, cause I'm not too sure, I seem to need `/proc r` as well.

[Ilya Dmitrichenko (errordeveloper)]

Regarding `/proc`, I does need to be `/proc r`, it fails without it. I still fails if use `@{PROC} r` or `@{PROC}/ r`, don't know what tat macro expands to and why there is so much of a difference. I can see that `/proc/ r` make it fail the same way too.

Jan 21 18:25:45 localhost.localdomain kernel: audit: type=1400 audit(1421864745.838:10): apparmor="DENIED" operation="open" profile="docker_docker_1.4.1.001" name="/proc" pid=1965 comm="dockerinit-1.4." requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Jamie Strandboge (jdstrand)]

Needing /proc is weird. /proc is a directory, not a file so the rule should be:

@{PROC}/ r,

However the denial very clearly says 'name="/proc"'. Is /proc not mounted at the time of the denial?

[Ilya Dmitrichenko (errordeveloper)]

No, it's certainly mounted.

[Jamie Strandboge (jdstrand)]

Hrm, this is probably because of the use of 'attach_disconnected'. Can you adjust /var/lib/apparmor/profiles/docker_docker_1.4.1.001 to remove '(attach_disconnected)', then try again, and paste the denial?

[Ilya Dmitrichenko (errordeveloper)]

I replaced `profile "docker_docker_1.4.1.001" (attach_disconnected) {` with `profile "docker_docker_1.4.1.001" {` and got:

Jan 22 07:27:58 localhost.localdomain kernel: audit: type=1400 audit(1421911678.548:33): apparmor="DENIED" operation="getattr" info="Failed name lookup - disconnected path" error=-13 profile="docker_docker_1.4.1.001" name="dev/null" pid=3929 comm="dockerinit-1.4." requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Jamie Strandboge (jdstrand)]

Sorry for just getting back to this now. Please feel free to add back the attach_disconnected and add '/proc' to the policy.