clustercheck container failed with Error response from daemon: invalid header field value \"oci runtime error: container_linux.go:247

Bug #1746298 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Unassigned

Bug Description

This failed in the tripleo gate:

 "Error running ['docker', 'run', '--name', 'clustercheck', '--label', 'config_id=tripleo_step2', '--label', 'container_name=clustercheck', '--label', 'managed_by=paunch', '--label', 'config_data={\"environment\": [\"KOLLA_CONFIG_STRATEGY=COPY_ALWAYS\", \"TRIPLEO_CONFIG_HASH=0ed049e3c2e0e1c92ba8bc5b004c7a6a\"], \"start_order\": 1, \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/kolla/config_files/clustercheck.json:/var/lib/kolla/config_files/config.json\", \"/var/lib/config-data/puppet-generated/clustercheck/:/var/lib/kolla/config_files/src:ro\", \"/var/lib/mysql:/var/lib/mysql\"], \"image\": \"192.168.24.1:8787/tripleomaster/centos-binary-mariadb:17a57d99b10f5872520e1a395214f80c9cfd1a65_c30c070f\", \"net\": \"host\", \"restart\": \"always\"}', '--detach=true', '--env=KOLLA_CONFIG_STRATEGY=COPY_ALWAYS', '--env=TRIPLEO_CONFIG_HASH=0ed049e3c2e0e1c92ba8bc5b004c7a6a', '--net=host', '--restart=always', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/kolla/config_files/clustercheck.json:/var/lib/kolla/config_files/config.json', '--volume=/var/lib/config-data/puppet-generated/clustercheck/:/var/lib/kolla/config_files/src:ro', '--volume=/var/lib/mysql:/var/lib/mysql', '192.168.24.1:8787/tripleomaster/centos-binary-mariadb:17a57d99b10f5872520e1a395214f80c9cfd1a65_c30c070f']. [127]",
            "stdout: 66566dfda314545974833d15647e1209710510404475a276d9cfd797f6944b1f",
            "stderr: /usr/bin/docker-current: Error response from daemon: invalid header field value \"oci runtime error: container_linux.go:247: starting container process caused \\\"process_linux.go:258: applying cgroup configuration for process caused \\\\\\\"open /sys/fs/cgroup/pids/system.slice/docker-66566dfda314545974833d15647e1209710510404475a276d9cfd797f6944b1f.scope/cgroup.procs: no such file or directory\\\\\\\"\\\"\\n\".",
            "Notice: Compiled catalog for centos-7-citycloud-sto2-0002270221.localdomain in environment production in 4.29 seconds",

http://logs.openstack.org/46/538346/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c0e6264/logs/undercloud/home/zuul/failed_deployments.log.txt.gz

http://logs.openstack.org/46/538346/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c0e6264/logs/undercloud/home/zuul/failed_deployment_list.log.txt.gz

http://logs.openstack.org/46/538346/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c0e6264/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-01-30_16_43_03

http://logs.openstack.org/46/538346/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c0e6264/logs/subnode-2/var/log/extra/docker/docker_allinfo.log.txt.gz

Tags: alert ci
Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Emilien Macchi (emilienm) wrote :

After some investigating, my theory is that oci-register-machine is not disabled by Puppet on time.

I can't find the "configure-oci-register-machine" File_line resource in the Puppet catalog:
http://logs.openstack.org/46/538346/1/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/c0e6264/logs/subnode-2/var/log/journal.txt.gz#_Jan_30_16_23_59

So maybe some later puppet run (not sure where) disable it but maybe too late and containers start with oci enabled...

Revision history for this message
Emilien Macchi (emilienm) wrote :
Revision history for this message
Alan Pevec (apevec) wrote :

This is the same as previously reported https://bugs.launchpad.net/tripleo/+bug/1744954 for periodic jobs.

Revision history for this message
Alan Pevec (apevec) wrote :

> disabled by default since...

That's included in oci-register-machine-0-5.13.git66691c3.el7_4 which is not published yet,
so let's see how we can force puppet to set this config early enough.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to puppet-tripleo (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/539537

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.