undercloud fails with " [ERROR] Could not lock /var/run/os-refresh-config.lock."

[Alfredo Moralejo]

We are hitting an intermittent issue in RDO-CI jobs using oooq where undercloud fails with following trace:

2017-03-01 14:52:54,672 INFO: dib-run-parts Wed Mar 1 14:52:54 UTC 2017 --------------------- END PROFILING ---------------------
2017-03-01 14:52:54,682 INFO: INFO: 2017-03-01 14:52:54,673 -- ############### End stdout/stderr logging ###############
2017-03-01 14:52:54,682 INFO: INFO: 2017-03-01 14:52:54,673 -- Running hook post-install
2017-03-01 14:52:54,682 INFO: INFO: 2017-03-01 14:52:54,673 -- Skipping hook post-install, the hook directory doesn't exist at /tmp/tmpbC7P9Y/post-install.d
2017-03-01 14:52:54,683 INFO: INFO: 2017-03-01 14:52:54,679 -- Ending run of instack.
2017-03-01 14:52:54,692 INFO: Instack completed successfully
2017-03-01 14:52:54,693 INFO: Running os-refresh-config
2017-03-01 14:52:54,814 INFO: [2017-03-01 14:52:54,814] (os-refresh-config) [ERROR] Could not lock /var/run/os-refresh-config.lock. [Errno 11] Resource temporarily unavailable
2017-03-01 14:52:54,823 ERROR:

We are seeing in ocata and master deployments. Logs for some of the failed jobs:

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-ocata-rdo_trunk-minimal_pacemaker-10/undercloud/home/stack/undercloud_install.log.gz

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-ocata-rdo_trunk-minimal_pacemaker-22/undercloud/home/stack/undercloud_install.log.gz

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-delorean-minimal_pacemaker-486/undercloud/home/stack/undercloud_install.log.gz

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-132/undercloud/home/stack/undercloud_install.log.gz

[wes hayutin (weshayutin)]

Issue is presenting itself again

https://thirdparty-logs.rdoproject.org/jenkins-tripleo-quickstart-master-delorean-baremetal-hp_dl360_envD-single_nic_vlans-180/undercloud/home/stack/undercloud_install.log.gz

[wes hayutin (weshayutin)]

sorry.. more specifically
https://thirdparty-logs.rdoproject.org/jenkins-tripleo-quickstart-master-delorean-baremetal-hp_dl360_envD-single_nic_vlans-180/undercloud/home/stack/undercloud_install.log.gz#_2017-06-29_12_37_59

[wes hayutin (weshayutin)]

showing up in our ci.centos gates again

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-gate-ocata-delorean-quick-basic-770/undercloud/home/stack/undercloud_install.log.gz

[Justin Kilpatrick (jkilpatr)]

also seeing this myself

https://thirdparty.logs.rdoproject.org/jenkins-browbeat-quickstart-gerrit-ocata-baremetal-CI-632/undercloud/home/stack/undercloud_install.log.txt.gz

[yatin (yatinkarel)]

Seen in RDO CI:
https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal_pacemaker-258/undercloud/home/stack/undercloud_install.log.gz
https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal-260/undercloud/home/stack/undercloud_install.log.gz

[Matt Young (halcyondude)]

Still being seen in rdophase2, pike

- https://trunk.rdoproject.org/centos7-pike/bb/ed/bbeda30a9df9021bf463b90d864e39cd6d7870df_b888d71c
- https://thirdparty.logs.rdoproject.org/jenkins-tripleo-quickstart-pike-rdo_trunk-baremetal-dell_pe_r630-bond_with_vlans-23/undercloud/home/stack/undercloud_install.log.txt.gz#_2017-08-31_18_23_43

[Matt Young (halcyondude)]

Still being seen. This time in rdophase1 (master)

- ad/bd/adbdd4ce7dbb10eaf1110c93f0c7fdee5f79b3c4_9f6039a1
- https://ci.centos.org/job/tripleo-quickstart-promote-master-current-tripleo-delorean-minimal_pacemaker/272
- https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-master-current-tripleo-delorean-minimal_pacemaker-272/undercloud/home/stack/undercloud_install.log.gz

[Matt Young (halcyondude)]

Added a patch for a recheck query here: https://review.openstack.org/#/c/527559/

[Ameed Ashour (ameeda)]

I faced the same issue , I could solve it by remove the file
$sudo rm /var/run/os-refresh-config.lock

what if I add script to check if the file exists, if exists delete it ?

[Moisés Guimarães de Medeiros (moguimar)]

Hitting this on a fresh tripleo-quickstart installation using the master-tripleo-ci release.

I see this in the journal logs:

Apr 26 10:05:44 undercloud os-collect-config[680]: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/ (Caused by ConnectTimeoutError(<requests.packages
Apr 26 10:05:44 undercloud os-collect-config[680]: Source [ec2] Unavailable.
Apr 26 10:05:44 undercloud os-collect-config[680]: Source [request] Unavailable.
Apr 26 10:05:44 undercloud os-collect-config[680]: /var/lib/os-collect-config/local-data not found. Skipping
Apr 26 10:05:44 undercloud os-collect-config[680]: No auth_url configured.
Apr 26 10:05:44 undercloud os-collect-config[680]: [2018-04-26 10:05:44,210] (os-refresh-config) [ERROR] Could not lock /var/run/os-refresh-config.lock. [Errno 11] Resource temporarily unavailable
Apr 26 10:05:44 undercloud os-collect-config[680]: Command failed, will not cache new data. Command 'os-refresh-config' returned non-zero exit status 11
Apr 26 10:05:57 undercloud os-collect-config[680]: HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /latest/meta-data/ (Caused by ConnectTimeoutError(<requests.packages
Apr 26 10:05:57 undercloud os-collect-config[680]: Source [ec2] Unavailable.
Apr 26 10:05:57 undercloud os-collect-config[680]: Source [request] Unavailable.
Apr 26 10:05:57 undercloud os-collect-config[680]: /var/lib/os-collect-config/local-data not found. Skipping

This is the end of the undercloud install logs:

2018-04-26 10:12:13 | 2018-04-26 10:12:13,496 INFO: + exit 1
2018-04-26 10:12:13 | 2018-04-26 10:12:13,502 INFO: [2018-04-26 10:12:13,496] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2018-04-26 10:12:13 | 2018-04-26 10:12:13,502 INFO:
2018-04-26 10:12:13 | 2018-04-26 10:12:13,502 INFO: [2018-04-26 10:12:13,496] (os-refresh-config) [ERROR] Aborting...
2018-04-26 10:12:13 | 2018-04-26 10:12:13,506 DEBUG: An exception occurred
2018-04-26 10:12:13 | Traceback (most recent call last):
2018-04-26 10:12:13 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2285, in install
2018-04-26 10:12:13 | _run_orc(instack_env)
2018-04-26 10:12:13 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1592, in _run_orc
2018-04-26 10:12:13 | _run_live_command(args, instack_env, 'os-refresh-config')
2018-04-26 10:12:13 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 671, in _run_live_command
2018-04-26 10:12:13 | raise RuntimeError('%s failed. See log for details.' % name)
2018-04-26 10:12:13 | RuntimeError: os-refresh-config failed. See log for details.
2018-04-26 10:12:13 | 2018-04-26 10:12:13,508 ERROR:
2018-04-26 10:12:13 | #############################################################################
2018-04-26 10:12:13 | Undercloud install failed.
2018-04-26 10:12:13 |
2018-04-26 10:12:13 | Reason: os-refresh-config failed. See log for details.
2018-04-26 10:12:13 |
2018-04-26 10:12:13 | See the previous output for detai...

[Ronelle Landy (rlandy)]

This problem still shows up sporadically.

[Martin Kopec (mkopec)]

Hitting the issue again in tripleo-ci-centos-7-scenario001-multinode-oooq:

http://logs.openstack.org/91/564291/22/check/tripleo-ci-centos-7-scenario001-multinode-oooq/a8eb702/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-10-11_09_44_50

[Bogdan Dobrelya (bogdando)]

We do not support instack undercloud for Stein, don't we?

[Marios Andreou (marios-b)]

@Bogdan the most recent trace from martin in comment #12 is pointing to ocata gate check and we do use it there https://review.openstack.org/#/c/564291/.

Having said that it seems we aren't seeing it any more that scenario job is green there. From the discussion and intermittent reports above it might be some race condition as it is not consistent/happens sporadically it seems. Eventually instack-undercloud _will_ go away but we will continue to use it until it goes end of life.

I'm not pushing back on wont fix for now, but i won't be surprised if we re-open it.

[Aakarsh (agopi)]

Saw this today in Perf-CI for Ocata build:

https://thirdparty.logs.rdoproject.org/jenkins-browbeat-quickstart-ocata-baremetal-mixed-82/undercloud/home/stack/undercloud_install.log.txt.gz#_2018-11-05_21_50_14