issue while installing docker inside s390x container

[Rajas Kakodkar]

This is in reference to https://github.com/multiarch/qemu-user-static/issues/108.
I am facing issue while installing docker inside s390x container under qemu on Ubuntu 18.04 host running on amd64.
Following are the contents of /proc/sys/fs/binfmt_misc/qemu-s390x on Intel host after running
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
enabled
interpreter /usr/bin/qemu-s390x-static
flags: F
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff
I could get docker service up with the following trick.
printf '{"iptables": false,"ip-masq": false,"bridge": "none" }' > /etc/docker/daemon.json
But even though docker service comes up, images are not getting pulled, docker pull fails with the following error.
failed to register layer: Error processing tar file(exit status 1):

[Laurent Vivier (laurent-vivier)]

Without a more detailled error message we can'know what happens.

What I see is you don't have the 'OC' flags that should help to execute binaries with (s) bit.

You should also report the version of qemu.

[Rajas Kakodkar (rajaskakodkar)]

The error message that I posted above is all that I see when I go to pull and image.
Following is the log from docker daemon

time="2020-05-26T07:34:31.385796553Z" level=info msg="API listen on /var/run/docker.sock"
time="2020-05-26T07:34:34.607431981Z" level=debug msg="Calling GET /_ping"
time="2020-05-26T07:34:34.635096021Z" level=debug msg="Calling GET /v1.38/containers/json"
time="2020-05-26T07:34:43.919894850Z" level=debug msg="Calling GET /_ping"
time="2020-05-26T07:34:43.963634642Z" level=debug msg="Calling GET /v1.38/info"
time="2020-05-26T07:34:44.702451808Z" level=debug msg="Calling POST /v1.38/images/create?fromImage=hello-world&tag=latest"
time="2020-05-26T07:34:44.715857621Z" level=debug msg="Trying to pull hello-world from https://registry-1.docker.io v2"
time="2020-05-26T07:34:46.805807639Z" level=debug msg="Pulling ref from V2 registry: hello-world:latest"
time="2020-05-26T07:34:46.806872106Z" level=debug msg="docker.io/library/hello-world:latest resolved to a manifestList object with 9 entries; looking for a unknown/s390x match"
time="2020-05-26T07:34:46.808815074Z" level=debug msg="found match for linux/s390x with media type application/vnd.docker.distribution.manifest.v2+json, digest sha256:e49abad529e5d9bd6787f3abeab94e09ba274fe34731349556a850b9aebbf7bf"
time="2020-05-26T07:34:47.233545931Z" level=debug msg="pulling blob \"sha256:3c80930bfdd5b53b7ca2a6b8116ed9a273af43a6b2dd13e81e82aae7521be469\""
time="2020-05-26T07:34:47.864739182Z" level=debug msg="Downloaded 3c80930bfdd5 to tempfile /var/lib/docker/tmp/GetImageBlob114078888"
time="2020-05-26T07:34:48.038875327Z" level=debug msg="Cleaning up layer b3da5d545c61d65059a2190feb19ae13797338ee999542b615e93e9708b88507: Error processing tar file(exit status 1): "
time="2020-05-26T07:34:48.049928529Z" level=info msg="Attempting next endpoint for pull after error: failed to register layer: Error processing tar file(exit status 1): "

Since I am using multiarch/qemu-user-static with the latest tag, qemu version is 4.2.0-6.

docker in docker works flawlessly on native s390x host. I am facing this issue only with qemu. Are there any other steps to follow to make docker in docker work under qemu?

[Nayana (nthorat)]

Any pointers about this issue? Any other steps needs to be followed to resolve the issue?

[Nayana (nthorat)]

Also same issue observed with docker version 19.03. docker pull command failing with an error "failed to register layer: Error processing tar file(exit status 1):"

[Thomas Huth (th-huth)]

QEMU 4.2 is quite old already, can you also reproduce the issue with the latest version of QEMU (v5.2 ... or maybe even with the 6.0-rc1 that should get released next week)?

[Nayana (nthorat)]

Yes we have observed the issue with multiarch/qemu-user-static image v5.2.0-1 and 5.2.0-2

[Nayana (nthorat)]

docker not starting inside s390x container. Log shows:

Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

[Nayana (nthorat)]

As we are facing issue with service docker start ( for docker in docker s390x container under qemu)

time="2021-03-23T07:29:07.645278866Z" level=warning msg="Running modprobe nf_nat failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
time="2021-03-23T07:29:07.645535879Z" level=warning msg="Running modprobe xt_conntrack failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"
Error starting daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.6.1: can't initialize iptables table `nat': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
 (exit status 3)

Does it mean that qemu doesn't support/have enabled related modules? modprobe?

[Nayana (nthorat)]

Could you please let me know if my understanding is correct? Qemu doesn't support/have enabled related modules? modprobe?

[Laurent Vivier (laurent-vivier)]

insmod/modprobe inside the container cannot load the modules because they are the one for the target architecture while the kernel is the one of the host. If you need functionalities in the container provided by some modules you need to load these modules using modprobe/insmod on the host.

[Nayana (nthorat)]

Looks like for docker in docker in cross architectures is not yet supported. if we use -v /var/run/docker.sock:/var/run/docker.sock , docker will always run as client in s390x container and server from amd64.

[Nayana (nthorat)]

Looks like docker in docker for cross architectures is not yet supported in qemu.
can be closed