Kubernetes test suite fails on mainline kernel 4.11+

[Chris Glass]

The to-be-introduced 4.13 based kernel fails the kubernetes test suite (https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gce-ubuntudev2-k8sbeta-default/15?log#log)

After rough bisecting (using kernels from http://kernel.ubuntu.com/~kernel-ppa/mainline/ ), it seems some kernel change introduced in 4.11 breaks those tests (4.10 is good, 4.11+ are bad. 4.11-rc* are bad).

The minimal reproducer (doesn't have to be on a cloud image - my bionic desktop reproduces it just as well):

sudo apt install docker.io
sudo docker run -d gcr.io/kubernetes-e2e-test-images/hostexec-amd64:1.0
(note down returned hash)
sudo docker exec -it (first few chars of returned hash) /bin/sh
# (Inside the docker prompt)
timeout -t 1 cat

On kernels exhibiting the problem, the "timeout/cat" command above will terminate the container.

On kernels exhibiting the problem, docker logs exhibits:

# sudo docker logs (first few chars of returned hash)
/bin/sh: can't open /fifo: Interrupted system call

Kernels that do not exhibit the problem will not terminate after the "timeout" command and return to a normal command prompt.

[Chris Glass (tribaal)]

Just tried with v4.15-rc7 from http://kernel.ubuntu.com/~kernel-ppa/mainline/ and the problem persists.

[Joseph Salisbury (jsalisbury)]

I started a kernel bisect between v4.10 final and v4.10-rc1. The kernel bisect will require testing of about 13 test kernels.

I built the first test kernel, up to the following commit:
caa59428971d5ad81d19512365c9ba580d83268c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1741887/caa59428971d5ad81d19512365c9ba580d83268c

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Chris Glass (tribaal)]

Kernel from #2 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/ca2dea434d10e3a676482fdf6df17d20cdb3e907

[Chris Glass (tribaal)]

Kernel from #4 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/5d8a00eee2ed2e548a5d21b0edf495f3f7bf8bb4

[Chris Glass (tribaal)]

kernel from #6 is bad.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/29a73d906bd386839015602c4bd35ef2e3531abc

[Chris Glass (tribaal)]

kernel from #8 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/ace71a19cec5eb430207c3269d8a2683f0574306

[Chris Glass (tribaal)]

kernel in #10 is bad.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/9b071c7935f053bf05ab4d0b2f4a8ae4069b4b89

[Chris Glass (tribaal)]

Kernel in #12 is good.

[Joseph Salisbury (jsalisbury)]

I built the first test kernel, up to the following commit:
601109c5c74a10e6b89465cb6aa31a40d1efc8e3

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1741887/601109c5c74a10e6b89465cb6aa31a40d1efc8e3

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

[Joseph Salisbury (jsalisbury)]

comment #14 should read the "Next" test kernel and not "first".

[Chris Glass (tribaal)]

Kernel in #14 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/f1ef09fde17f9b77ca1435a5b53a28b203afb81c

[Chris Glass (tribaal)]

Kernel in #17 is bad

[Joseph Salisbury (jsalisbury)]

f1ef09fde17f9b77ca1435a5b53a28b203afb81c was good.

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/d6cffbbe9a7e51eb705182965a189457c17ba8a3

[Chris Glass (tribaal)]

Kernel in #19 is bad.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/c6c70f4455d1eda91065e93cc4f7eddf4499b105

[Chris Glass (tribaal)]

Kernel in #21 is bad

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/20523132ec5d1b481e1d66557292ed3a3021e817

[Chris Glass (tribaal)]

Kernel in #23 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/68eb94f16227336a5773b83ecfa8290f1d6b78ce

[Chris Glass (tribaal)]

Kernel in #25 is good.

[Joseph Salisbury (jsalisbury)]

Next kernel is ready:

http://kernel.ubuntu.com/~jsalisbury/lp1741887/1328c727004d432bbdfba0ffa02a166df04c7305

[Chris Glass (tribaal)]

Kernel in #27 was good.

[Chris Glass (tribaal)]

Looks like the failure there is due to the following commit in the linux kernel:

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=c6c70f4455d1eda91065e93cc4f7eddf4499b105

[Joseph Salisbury (jsalisbury)]

I noticed this bug was still open. I built a test kernel with a revert of commit c6c70f4455d.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1741887

Can you test this kernel and see if it resolves this bug?

It might also be worth testing the v4.16-rc4 mainline kernel to see if a fix already exists:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16-rc4/