unable to find user root: no matching entries in passwd file

[Cédric Jeanneret]

We sometime hit the following issue while deploying an undercloud:

2018-11-13 18:16:19 | "2018-11-13 18:13:39,017 ERROR: 4514 -- Failed running docker-puppet.py for heat_api",
2018-11-13 18:16:19 | "2018-11-13 18:13:39,018 ERROR: 4514 -- error mounting image volumes: unable to find user root: no matching entries in passwd file",

Problem is: it's not always the case, and it's not always the same container. Until now, I had it for swift-related and heat-related containers.

It's happening for a while, at random:
http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22no%20matching%20entries%20in%20passwd%20file%5C%22

[Cédric Jeanneret (cjeanner)]

Issue on github: https://github.com/containers/libpod/issues/1817

[Rabi Mishra (rabi)]

Possibly a runc issue as there is some discussion[1] about something similar in a docker thread[1] which ends with:

"Currently there is no resolution to this issue. It is being tracked via an internal bug.”

And that's not good:/

[1] https://forums.docker.com/t/unable-to-find-user-root-no-matching-entries-in-passwd-file/26545

[Harald Jensås (harald-jensas)]

This cause the gate job to fail:

http://logs.openstack.org/43/613043/3/gate/tripleo-ci-centos-7-containers-multinode/842fffd/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-29_05_58_22

[Marios Andreou (marios-b)]

just seen this on the master tht gate check centos standalone http://logs.openstack.org/98/604298/105/check/tripleo-ci-centos-7-standalone/5b9e669/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-11-29_20_50_50

 2018-11-29 20:50:50 | "2018-11-29 20:48:29,700 ERROR: 20487 -- error mounting image volumes: unable to find user root: no matching entries in passwd file",

[Sorin Sbarnea (ssbarnea)]

We are seeing more and more of these errors and they seems to be caused by "classified" docker bug.

http://status.openstack.org/elastic-recheck/index.html#1803544

https://forums.docker.com/t/unable-to-find-user-root-no-matching-entries-in-passwd-file/26545/7

What can we do about it?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/624420

[wes hayutin (weshayutin)]

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22error%20mounting%20image%20volumes%3A%20unable%20to%20find%20user%20root%3A%20no%20matching%20entries%20in%20passwd%20file%5C%22%20AND%20tags%3A%5C%22console%5C%22

[wes hayutin (weshayutin)]

This is happening a lot and failing the gate, I see it's being worked on but setting alert for visibility.

[Emilien Macchi (emilienm)]

https://github.com/containers/libpod/pull/1978 should help

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/624420

[Rabi Mishra (rabi)]

I guess we've not got podman with the fix yet as I still see the issue.

https://logs.rdoproject.org/99/624999/2/openstack-check/tripleo-ci-centos-7-multinode-1ctlr-featureset010/574230e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-12-14_04_31_53

[Sorin Sbarnea (ssbarnea)]

The fix is supposed to land before the end of the week. I am sure that Emilien with post a comment when this lands.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/624420

[wes hayutin (weshayutin)]

Bug 1803544 - unable to find user root: no matching entries in passwd file
0 fails in 24 hrs / 0 fails in 10 days
Projects: (tripleo - Triaged)
No matches
Logstash Launchpad

[wes hayutin (weshayutin)]

closing.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696120
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=393e96b5b9e8affd35b48fb45f0a75b253629074
Submitter: Zuul
Branch: master

commit 393e96b5b9e8affd35b48fb45f0a75b253629074
Author: Michele Baldessari <email address hidden>
Date: Tue Nov 26 17:18:03 2019 +0100

    Use '0' instead of root in container-puppet.py

    Even though the number of user lookups have been reduced from two to one
    via https://github.com/containers/libpod/pull/1978, we still see the
    following error from time to time:
    time="2019-11-22T19:19:33Z" level=debug msg="ExitCode msg: \"unable to find user root: no matching entries in passwd file\""
    time="2019-11-22T19:19:33Z" level=error msg="unable to find user root: no matching entries in passwd file"

    The TLDR; is that podman/docker, when passed a --user=<name> parameter,
    will parse the /etc/passwd file inside the container and detect the
    uid/gid to switch to. The problem seems to be that sometimes this
    /etc/passwd is either read as empty or non-existant when we try and
    parse it (the root-cause of which is the real underlying bug).

    Since it seems that root-causing this will take a rather large amount of
    time, we can just pass the UID directly which will not fail when
    the parsing code cannot find the specified user in /etc/passwd, as it
    simply uses the provided UID:
    https://github.com/containers/libpod/blob/master/vendor/github.com/opencontainers/runc/libcontainer/user/user.go#L333

    Tested this by running a reproducer on three machines for a total
    of ~800 runs and had 0 occurrences of this error. Previously I could
    reproduce this issue in about 30 to 60 runs at most.

    Related rhbz: 1776766
    Related-Bug: #1803544

    Change-Id: Ia9860107c35e543a05775596076873ea950b7400

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/696641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696641
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=871c1a30326c3c5b0e3703a0ab64deedc1ded5a6
Submitter: Zuul
Branch: stable/train

commit 871c1a30326c3c5b0e3703a0ab64deedc1ded5a6
Author: Michele Baldessari <email address hidden>
Date: Tue Nov 26 17:18:03 2019 +0100

    Use '0' instead of root in container-puppet.py

    Even though the number of user lookups have been reduced from two to one
    via https://github.com/containers/libpod/pull/1978, we still see the
    following error from time to time:
    time="2019-11-22T19:19:33Z" level=debug msg="ExitCode msg: \"unable to find user root: no matching entries in passwd file\""
    time="2019-11-22T19:19:33Z" level=error msg="unable to find user root: no matching entries in passwd file"

    The TLDR; is that podman/docker, when passed a --user=<name> parameter,
    will parse the /etc/passwd file inside the container and detect the
    uid/gid to switch to. The problem seems to be that sometimes this
    /etc/passwd is either read as empty or non-existant when we try and
    parse it (the root-cause of which is the real underlying bug).

    Since it seems that root-causing this will take a rather large amount of
    time, we can just pass the UID directly which will not fail when
    the parsing code cannot find the specified user in /etc/passwd, as it
    simply uses the provided UID:
    https://github.com/containers/libpod/blob/master/vendor/github.com/opencontainers/runc/libcontainer/user/user.go#L333

    Tested this by running a reproducer on three machines for a total
    of ~800 runs and had 0 occurrences of this error. Previously I could
    reproduce this issue in about 30 to 60 runs at most.

    Related rhbz: 1776766
    Related-Bug: #1803544

    NB: Cherry-pick not 100% clean

    Change-Id: Ia9860107c35e543a05775596076873ea950b7400
    (cherry picked from commit 393e96b5b9e8affd35b48fb45f0a75b253629074)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/720588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/720588
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=cea234d40ef70a740421759771b97c5b20aac8d5
Submitter: Zuul
Branch: stable/stein

commit cea234d40ef70a740421759771b97c5b20aac8d5
Author: Michele Baldessari <email address hidden>
Date: Tue Nov 26 17:18:03 2019 +0100

    Use '0' instead of root in container-puppet.py

    Even though the number of user lookups have been reduced from two to one
    via https://github.com/containers/libpod/pull/1978, we still see the
    following error from time to time:
    time="2019-11-22T19:19:33Z" level=debug msg="ExitCode msg: \"unable to find user root: no matching entries in passwd file\""
    time="2019-11-22T19:19:33Z" level=error msg="unable to find user root: no matching entries in passwd file"

    The TLDR; is that podman/docker, when passed a --user=<name> parameter,
    will parse the /etc/passwd file inside the container and detect the
    uid/gid to switch to. The problem seems to be that sometimes this
    /etc/passwd is either read as empty or non-existant when we try and
    parse it (the root-cause of which is the real underlying bug).

    Since it seems that root-causing this will take a rather large amount of
    time, we can just pass the UID directly which will not fail when
    the parsing code cannot find the specified user in /etc/passwd, as it
    simply uses the provided UID:
    https://github.com/containers/libpod/blob/master/vendor/github.com/opencontainers/runc/libcontainer/user/user.go#L333

    Tested this by running a reproducer on three machines for a total
    of ~800 runs and had 0 occurrences of this error. Previously I could
    reproduce this issue in about 30 to 60 runs at most.

    Related rhbz: 1776766
    Related-Bug: #1803544

    NB: Cherry-pick not 100% clean

    Change-Id: Ia9860107c35e543a05775596076873ea950b7400
    (cherry picked from commit 393e96b5b9e8affd35b48fb45f0a75b253629074)
    (cherry picked from commit 871c1a30326c3c5b0e3703a0ab64deedc1ded5a6)