CI: containers job fails with "oci runtime error..."

[Sagi (Sergey) Shnaidman]

CI containers job fails a lot recently with error when deploying overcloud:

http://logs.openstack.org/38/462538/3/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/e5d3dee/logs/oooq/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2017-05-22_03_44_29

Digest: sha256:b8924c9b47b12abd09c25b188e2000d024a07ce3bf12fb65dfae34ebc7596d39
2017-05-22 03:44:29 | /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/config-data/memcached/etc/sysconfig/memcached\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/53a9bcec3eeee7bc5215cb0cfefa604a371cdac976d9f647004ea7617f044a3d/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/53a9bcec3eeee7bc5215cb0cfefa604a371cdac976d9f647004ea7617f044a3d/rootfs/etc/sysconfig/memcached\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n".
2017-05-22 03:44:29 |
2017-05-22 03:44:29 | /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/var/lib/config-data/memcached/etc/sysconfig/memcached\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/020507c7c83eaa7550fc619c6328385a08406eb6cd68a206c9cbecaf99f48560/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/020507c7c83eaa7550fc619c6328385a08406eb6cd68a206c9cbecaf99f48560/rootfs/etc/sysconfig/memcached\\\\\\\" causeHeat Stack create failed.
2017-05-22 03:44:29 | Heat Stack create failed.
2017-05-22 03:44:29 | d \\\\\\\"not a directory\\\\\\\"\\\"\"\n".
2017-05-22 03:44:29 | (truncated, view all with --long)
2017-05-22 03:44:29 | overcloud.AllNodesDeploySteps.ControllerGenerateConfigDeployment:
2017-05-22 03:44:29 | resource_type: OS::Heat::SoftwareDeploymentGroup
2017-05-22 03:44:29 | physical_resource_id: a2d9c18e-7a80-4a5b-a3bf-f0c597615db7
2017-05-22 03:44:29 | status: CREATE_FAILED
2017-05-22 03:44:29 | status_reason: |
2017-05-22 03:44:29 | CREATE aborted

[Bogdan Dobrelya (bogdando)]

interesting, why it mentions /var/lib/docker/devicemapper/mnt after the overlay2 switch done by https://review.openstack.org/#/c/451916/ ?

[Bogdan Dobrelya (bogdando)]

Looks related https://bugs.launchpad.net/tripleo/+bug/1692502
Could the docker setup then end up being a half overlay2/devicemapper split somehow?

[Jiří Stránský (jistr)]

This is intermittent, some job overcloud container jobs are passing. (Could be a race condition of some sort?)

[Jiří Stránský (jistr)]

Both successful and failed jobs still seem to use devicemapper+loopback file as the storage backend:

http://logs.openstack.org/90/467490/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/38c87f4/logs/oooq/overcloud-controller-0/var/log/extra/docker/docker_allinfo.log.txt.gz

http://logs.openstack.org/84/467484/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/ceaf586/logs/oooq/overcloud-controller-0/var/log/extra/docker/docker_allinfo.log.txt.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/467572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467572
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a42b93d503690cb56d6a1053759f5b9a9a65fd4e
Submitter: Jenkins
Branch: master

commit a42b93d503690cb56d6a1053759f5b9a9a65fd4e
Author: Jiri Stransky <email address hidden>
Date: Wed May 24 12:21:47 2017 +0200

    Fix race condition between docker-puppet and container steps

    GenerateConfigDeployment wasn't anchored with dependencies anywhere. If
    it took too long to complete and step 1 of containers creation already
    started executing, problems happened. This is now fixed by adding the
    required dependency relationship.

    Change-Id: Ie7dfd2a965e704ba278d4c2fad67f14a3a62799e
    Closes-Bug: #1692503

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.