To confirm that this isn't shiftfs related and that we were just causing the issue to be hidden, I've run the same test on OpenSuse tumbleweed.
I chose that distro because it's apparmor-enabled, has snapd and a 5.4 kernel.
```
localhost:~ # snap install docker
docker 18.09.9 from Canonical* installed
localhost:~ # auth_folder=/var/snap/docker/common/auth
localhost:~ # mkdir -p $auth_folder
localhost:~ # docker run --entrypoint htpasswd registry:2 -Bbn user passwd > $auth_folder/htpasswd
Unable to find image 'registry:2' locally
2: Pulling from library/registry
486039affc0a: Pulling fs layer
ba51a3b098e6: Pulling fs layer
8bb4c43d6c8e: Pulling fs layer
6f5f453e5f2d: Pulling fs layer
42bc10b72f42: Pulling fs layer
6f5f453e5f2d: Waiting
42bc10b72f42: Waiting
ba51a3b098e6: Download complete
486039affc0a: Verifying Checksum
486039affc0a: Download complete
8bb4c43d6c8e: Verifying Checksum
8bb4c43d6c8e: Download complete
6f5f453e5f2d: Verifying Checksum
6f5f453e5f2d: Download complete
42bc10b72f42: Verifying Checksum
42bc10b72f42: Download complete
486039affc0a: Pull complete
ba51a3b098e6: Pull complete
8bb4c43d6c8e: Pull complete
6f5f453e5f2d: Pull complete
42bc10b72f42: Pull complete
Digest: sha256:7d081088e4bfd632a88e3f3bcd9e007ef44a796fddfe3261407a3f9f04abe1e7
Status: Downloaded newer image for registry:2
localhost:~ # docker run -d -p 5000:5000 --restart=always --name registry \
> -v $auth_folder:/auth \
> -e "REGISTRY_AUTH=htpasswd" \
> -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
> -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
> registry:2
cba1ec94734a8a198fa0c474d9873233958fad6cdafe93d2ccf4d701ecab55ff
localhost:~ # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cba1ec94734a registry:2 "/entrypoint.sh /etc…" 5 seconds ago Restarting (2) Less than a second ago registry
localhost:~ # uname -a
Linux localhost 5.4.10-1-default #1 SMP Thu Jan 9 15:45:45 UTC 2020 (556a6fe) x86_64 x86_64 x86_64 GNU/Linux
localhost:~ #
```
As you can see, the exact same thing happen there. So this is an apparmor kernel bug or some issue with the snapd or docker snap, this isn't a shiftfs bug and reverting the change would just expose a different bug rather than actually fix things.
To confirm that this isn't shiftfs related and that we were just causing the issue to be hidden, I've run the same test on OpenSuse tumbleweed.
I chose that distro because it's apparmor-enabled, has snapd and a 5.4 kernel.
``` /var/snap/ docker/ common/ auth htpasswd 7d081088e4bfd63 2a88e3f3bcd9e00 7ef44a796fddfe3 261407a3f9f04ab e1e7 AUTH=htpasswd" \ AUTH_HTPASSWD_ REALM=Registry Realm" \ AUTH_HTPASSWD_ PATH=/auth/ htpasswd \ 98fa0c474d98732 33958fad6cdafe9 3d2ccf4d701ecab 55ff
localhost:~ # snap install docker
docker 18.09.9 from Canonical* installed
localhost:~ # auth_folder=
localhost:~ # mkdir -p $auth_folder
localhost:~ # docker run --entrypoint htpasswd registry:2 -Bbn user passwd > $auth_folder/
Unable to find image 'registry:2' locally
2: Pulling from library/registry
486039affc0a: Pulling fs layer
ba51a3b098e6: Pulling fs layer
8bb4c43d6c8e: Pulling fs layer
6f5f453e5f2d: Pulling fs layer
42bc10b72f42: Pulling fs layer
6f5f453e5f2d: Waiting
42bc10b72f42: Waiting
ba51a3b098e6: Download complete
486039affc0a: Verifying Checksum
486039affc0a: Download complete
8bb4c43d6c8e: Verifying Checksum
8bb4c43d6c8e: Download complete
6f5f453e5f2d: Verifying Checksum
6f5f453e5f2d: Download complete
42bc10b72f42: Verifying Checksum
42bc10b72f42: Download complete
486039affc0a: Pull complete
ba51a3b098e6: Pull complete
8bb4c43d6c8e: Pull complete
6f5f453e5f2d: Pull complete
42bc10b72f42: Pull complete
Digest: sha256:
Status: Downloaded newer image for registry:2
localhost:~ # docker run -d -p 5000:5000 --restart=always --name registry \
> -v $auth_folder:/auth \
> -e "REGISTRY_
> -e "REGISTRY_
> -e REGISTRY_
> registry:2
cba1ec94734a8a1
localhost:~ # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cba1ec94734a registry:2 "/entrypoint.sh /etc…" 5 seconds ago Restarting (2) Less than a second ago registry
localhost:~ # uname -a
Linux localhost 5.4.10-1-default #1 SMP Thu Jan 9 15:45:45 UTC 2020 (556a6fe) x86_64 x86_64 x86_64 GNU/Linux
localhost:~ #
```
As you can see, the exact same thing happen there. So this is an apparmor kernel bug or some issue with the snapd or docker snap, this isn't a shiftfs bug and reverting the change would just expose a different bug rather than actually fix things.