Containerized undercloud fails to mount /etc/ssh/ssh_known_hosts when /etc/ is mounted readonly

[Oliver Walsh]

I hit this while trying to land https://review.openstack.org/468461.

Undercloud docker config for ironic-api dbsync fails to mount /etc/ssh/ssh_known_hosts when /var/lib/config-data/ironic/etc/ is mount on /etc as readonly.

(http://logs.openstack.org/61/468461/2/check/gate-tripleo-ci-centos-7-undercloud-containers-nv/d6d4fd3/logs/var/log/undercloud_install.txt.gz)

2017-06-06 12:19:56.000 | [2017-06-06 12:19:18,579] (heat-config) [DEBUG] docker run --name ironic_db_sync --label deploy_stack_id=undercloud-AllNodesDeploySteps-62l25vo6qe6d-UndercloudContainersDeployment_Step3-rec6mxppkozq/faa604ed-a4f4-4666-bd1b-eb0927ab31ca --label deploy_resource_name=0 --label config_id=d608d7e6-e336-457e-93a0-a6917db2632c --label container_name=ironic_db_sync --label managed_by=docker-cmd --net=host --privileged=false --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro --volume=/var/lib/config-data/ironic/etc/:/etc/:ro --volume=/var/log/containers/ironic:/var/log/ironic tripleoupstream/centos-binary-ironic-api:latest /usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c 'ironic-dbsync --config-file /etc/ironic/ironic.conf'
2017-06-06 12:19:56.000 | [2017-06-06 12:19:18,946] (heat-config) [DEBUG]
2017-06-06 12:19:56.000 | [2017-06-06 12:19:18,946] (heat-config) [DEBUG] container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/etc/ssh/ssh_known_hosts\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs\\\" at \\\"/var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs/etc/ssh/ssh_known_hosts\\\" caused \\\"open /var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs/etc/ssh/ssh_known_hosts: read-only file system\\\"\""
2017-06-06 12:19:56.000 | /usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/etc/ssh/ssh_known_hosts\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs\\\\\\\" at \\\\\\\"/var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs/etc/ssh/ssh_known_hosts\\\\\\\" caused \\\\\\\"open /var/lib/docker/devicemapper/mnt/08c063a6399ab89c5fea820c47aacf930df2198d736794c0d441b6388b75c592/rootfs/etc/ssh/ssh_known_hosts: read-only file system\\\\\\\"\\\"\"\n".

Yet it works ok when /etc/puppet/ and /etc/hosts are mounted:

(http://logs.openstack.org/47/471447/1/check/gate-tripleo-ci-centos-7-undercloud-containers-nv/087976e/logs/var/log/undercloud_install.txt.gz)

2017-06-06 19:43:40.000 | [2017-06-06 19:42:26,339] (heat-config) [DEBUG] docker run --name ironic_db_sync --label deploy_stack_id=undercloud-AllNodesDeploySteps-htct6tmz4mae-UndercloudContainersDeployment_Step3-nt4th3yf7mwc/1d9bd2b5-b087-40da-bf56-3628adee26b7 --label deploy_resource_name=0 --label config_id=d50a0b43-d864-417e-8a31-c2dddb52d008 --label container_name=ironic_db_sync --label managed_by=docker-cmd --net=host --privileged=false --user=root --volume=/etc/hosts:/etc/hosts:ro --volume=/etc/localtime:/etc/localtime:ro --volume=/etc/puppet:/etc/puppet:ro --volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume=/dev/log:/dev/log --volume=/var/lib/config-data/ironic/etc/:/etc/:ro --volume=/var/log/containers/ironic:/var/log/ironic tripleoupstream/centos-binary-ironic-api:latest /usr/bin/bootstrap_host_exec ironic_api su ironic -s /bin/bash -c 'ironic-dbsync --config-file /etc/ironic/ironic.conf'

From the various permutations I've tried, either removing the /etc/ssh_known_hosts or removing the /etc mounts resolves this. I suspect it's a docker/devicemapper issue. Raising an LP bug as a reminder to look into this later.

[Oliver Walsh (owalsh)]

Reproduced with overlay2:

/usr/bin/docker-current: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:54: mounting \\\\\\\"/etc/ssh/ssh_known_hosts\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/b9c3a786f30dfda5d961dfe74998c8edc09dee4a95c83b4d610babd13198e625/merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/b9c3a786f30dfda5d961dfe74998c8edc09dee4a95c83b4d610babd13198e625/merged/etc/ssh/ssh_known_hosts\\\\\\\" caused \\\\\\\"open /var/lib/docker/overlay2/b9c3a786f30dfda5d961dfe74998c8edc09dee4a95c83b4d610babd13198e625/merged/etc/ssh/ssh_known_hosts: read-only file system\\\\\\\"\\\"\"\n".

[Oliver Walsh (owalsh)]

Appears to be this particular file that causes problems:

()[root@overcloud-novacompute-1 /]# touch /etc/ssh/ssh_known_hosts
touch: cannot touch ‘/etc/ssh/ssh_known_hosts’: Read-only file system

[Oliver Walsh (owalsh)]

/var/lib/config_data/<foo>/ssh/ssh_known_hosts does not exist. If I create it then the error goes away.

[Oliver Walsh (owalsh)]

Thinking about it this is expected. Adding a new file to /etc/ssh would need to modify the directory entry which is in the /etc/:ro mount.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/471475
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e7b7d4a0e3cda477324553df74ed1c98985a8a88
Submitter: Jenkins
Branch: master

commit e7b7d4a0e3cda477324553df74ed1c98985a8a88
Author: Oliver Walsh <email address hidden>
Date: Tue Jun 6 21:16:38 2017 +0100

    Ensure /etc/ssh/ssh_known_hosts exist in docker config-data.

    Works around the issue encountered in 1696283.

    Change-Id: I1947d9d1e3cabc5dfe25ee1af994d684425bdbf7
    Resolves-Bug: #1696283

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/478918

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/478918
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=23a6040d6a45f6c05301c9ef860646eb2678a60c
Submitter: Jenkins
Branch: master

commit 23a6040d6a45f6c05301c9ef860646eb2678a60c
Author: Martin André <email address hidden>
Date: Thu Jun 29 14:34:52 2017 +0200

    Create /etc/ssh/ssh_known_hosts on base image

    Move the workaround for bug #1696283 from docker-puppet.py to the
    containers' base image, as it is incompatible with the way we want to
    copy all generated config files in the containers [1]. Indeed, the
    /etc/ssh/ssh_known_hosts is mounted read-only in the containers [2] and
    kolla set_configs fails to replace it if it exists in the config files.

    [1] Patch Ia4ad6d77387e3dc354cd131c2f9756939fb8f736
    [2] https://github.com/openstack/tripleo-heat-templates/blob/3b7de8a2514cf9b90ee7aec3365bfbf4547175ad/docker/services/containers-common.yaml#L62

    Change-Id: I46e860b487882cec285e131174841c85a2ac9b1a
    Related-Bug: #1696283