sshd stops reading puppet output on applying keystone.pp during fuel-qa auto tests
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Fuel for OpenStack |
High
|
Nikita Karpin | ||
Bug Description
Detailed bug description:
For some reason puppet stuck on applying /etc/puppet/
At the same time manually applying works fine.
Failed systest example https:/
As you can see, systest failed due to timeout.
Step to reproduce:
Do any change in any _centos_ package and send it to review.
In my case even version change was enough.
Expected results:
Systest will success.
Actual results:
Systest failed.
Reproducibility:
Always.
| Ivan Suzdal (isuzdal) wrote : | #1 |
| Changed in fuel: | |
| status: | New → Confirmed |
| milestone: | none → 11.0 |
| tags: | added: area-puppet |
| Nikita Karpin (mkarpin) wrote : | #2 |
| Changed in fuel: | |
| status: | Confirmed → In Progress |
| Nikita Karpin (mkarpin) wrote : | #3 |
the problem is not in manifests or puppet or keystone, it is some magic between fuel-devops/qa + bash + puppet output, because when I redirected puppet logging to syslog, update-
Related fix proposed to branch: master
Review: https:/
We found that the reason of puppet being stuck is hanging of sshd while reading output from update-
1) ps ouput with stucked sshd fork (pid 11866):
06:root 11866 0.1 0.0 136812 2404 ? Ss 09:16 0:03 sshd: root@notty
107-root 11892 0.0 0.0 52700 792 ? Ss 09:16 0:00 \_ /usr/libexec/
108-root 17818 0.0 0.0 113128 1468 ? Ss 09:33 0:00 \_ /bin/bash /usr/share/
109-root 17825 0.0 0.0 113124 824 ? S 09:33 0:00 \_ /bin/bash /usr/share/
110-root 17827 0.0 0.0 107896 664 ? S 09:33 0:00 | \_ tee -i /var/log/
111-root 17867 0.0 0.0 113128 1436 ? S 09:33 0:00 \_ bash -x /etc/puppet/
112-root 24787 0.1 1.8 749696 53604 ? Sl 09:35 0:02 \_ /usr/bin/ruby /usr/bin/puppet apply -d -v --color false --detailed-
2) There are file descriptors opened by sshd:
ls -la /proc/11866/fd/
total 0
dr-x------. 2 root root 0 лют 23 09:31 .
dr-xr-xr-x. 9 root root 0 лют 23 09:16 ..
lrwx------. 1 root root 64 лют 23 09:31 0 -> /dev/null
lrwx------. 1 root root 64 лют 23 09:31 1 -> /dev/null
lr-x------. 1 root root 64 лют 23 09:31 10 -> pipe:[34793]
l-wx------. 1 root root 64 лют 23 09:35 11 -> pipe:[204872]
lr-x------. 1 root root 64 лют 23 09:31 12 -> pipe:[34794]
lr-x------. 1 root root 64 лют 23 09:35 13 -> pipe:[204873]
lr-x------. 1 root root 64 лют 23 09:35 15 -> pipe:[204874]
lrwx------. 1 root root 64 лют 23 09:31 2 -> /dev/null
lrwx------. 1 root root 64 лют 23 09:31 3 -> socket:[34621]
lrwx------. 1 root root 64 лют 23 09:31 4 -> socket:[34681]
lr-x------. 1 root root 64 лют 23 09:31 5 -> pipe:[34687]
l-wx------. 1 root root 64 лют 23 09:31 6 -> /run/systemd/
l-wx------. 1 root root 64 лют 23 09:31 7 -> pipe:[34687]
l-wx------. 1 root root 64 лют 23 09:31 9 -> pipe:[34792]
FD 13 is descriptor of pipe connected to update-
FD 3 is socket of ssh connection
3) netstat output:
tcp 0 0 10.109.35.2:22 10.109.35.1:50157 ESTABLISHED
lsof -n | grep 34621
sshd 11866 root 3u IPv4 34621 0t0 TCP 10.109.
4) Fragment of strace of sshd:
http://
As you can see for some reason FD13 disappeared from SSHD select() syscall. Compare lines 4 and 17 of the above fragment. The exact reason of such behavior is still unclear. So we decided to fix this in fuel-qa code.
Fix proposed to branch: master
Review: https:/
| summary: |
- Puppet stuck on applying keystone_token_disable.pp + sshd stops reading puppet output on applying keystone.pp |
| summary: |
- sshd stops reading puppet output on applying keystone.pp + sshd stops reading puppet output on applying keystone.pp during fuel-qa + auto tests |
Change abandoned by Mykyta Karpin (<email address hidden>) on branch: master
Review: https:/
Reason: in favor of https:/
| Changed in fuel: | |
| status: | In Progress → Fix Committed |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit ee0bf1cfa424aca
Author: Mykyta Karpin <email address hidden>
Date: Thu Feb 23 15:22:10 2017 +0200
Redirect update-
SSHD stops reading of update-
during puppet run, this causes puppet hanging on
keystone tasks. In order to avoid this we need to
redirect script's output to /dev/null.
Change-Id: I99959cb72caeec
Closes-Bug: #1664635
Fix proposed to branch: stable/ocata
Review: https:/
Fix proposed to branch: stable/newton
Review: https:/
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/newton
commit b431c985530508e
Author: Mykyta Karpin <email address hidden>
Date: Thu Feb 23 15:22:10 2017 +0200
Redirect update-
SSHD stops reading of update-
during puppet run, this causes puppet hanging on
keystone tasks. In order to avoid this we need to
redirect script's output to /dev/null.
Change-Id: I99959cb72caeec
Closes-Bug: #1664635
(cherry picked from commit ee0bf1cfa424aca
| tags: | added: in-stable-newton |
| tags: | added: in-stable-ocata |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/ocata
commit cb4a6873534c810
Author: Mykyta Karpin <email address hidden>
Date: Thu Feb 23 15:22:10 2017 +0200
Redirect update-
SSHD stops reading of update-
during puppet run, this causes puppet hanging on
keystone tasks. In order to avoid this we need to
redirect script's output to /dev/null.
Change-Id: I99959cb72caeec
Closes-Bug: #1664635
(cherry picked from commit ee0bf1cfa424aca
This issue was fixed in the openstack/fuel-qa 11.0.0.0rc2 release candidate.
trying to strace puppet run, it hangs on
/usr/bin/python2 /usr/bin/openstack user show --format shell monitord --domain 058cf5ffa0e9418
trying to find the cause...