[Reduced footprint] Compute service is disabled after reboot `virt, compute` node

[Ksenia Svechnikova]

FUEL 8.0 RC1

Steps:

1. On HW lab deploy VLAN env with 3xcompute,virt node with 3xKVMxcontrollers

[root@fuel-support4-mos8 ~]# fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online | group_id
---|----------|-----------------------------|---------|---------------|-------------------|---------------|---------------|--------|---------
1 | ready | cz7394-kvm.host-telecom.com | 1 | 172.16.58.201 | 0c:c4:7a:13:4a:90 | compute, virt | | True | 1
3 | ready | cz7395-kvm.host-telecom.com | 1 | 172.16.58.202 | 0c:c4:7a:13:4c:88 | compute, virt | | True | 1
9 | ready | Untitled (49:04) | 1 | 172.16.58.209 | 52:54:00:7f:49:04 | controller | | True | 1
4 | discover | cz7396-kvm.host-telecom.com | None | 172.16.58.203 | 0c:c4:7a:15:ed:46 | | | True | None
10 | ready | Untitled (a0:35) | 1 | 172.16.58.208 | 52:54:00:6c:a0:35 | controller | | True | 1
2 | ready | cz7393-kvm.host-telecom.com | 1 | 172.16.58.200 | 0c:c4:7a:15:ec:3c | compute, virt | | True | 1
11 | ready | Untitled (ff:32) | 1 | 172.16.58.207 | 52:54:00:95:ff:32 | controller | | True | 1

2. Reboot comute, virt node (node-2 + node-10)

After the reboot, nova-compute service is disabled with error in nova-compute: [1]

<179>Feb 10 06:32:04 node-2 nova-compute: 2016-02-10 06:32:04.948 4820 ERROR nova.virt.libvirt.host [req-1cebaea1-79b5-4111-8a93-6943d238957a - - - - -] Connection to libvirt failed: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

Libvirt error: [2]

So nova-compute service try to start before the ovs port is created

[1] http://paste.openstack.org/show/486520/
[2] http://paste.openstack.org/show/486521/

Workaround:

1. Wait a period of time (5-10 min) till the socket /var/run/libvirt/libvirt-sock will be created on compute
2. Restart nova-compute service on node-2:

service nova-compute restart

[slava valyavskiy (slava-val-al)]

Moreover, libvirt is not able to communicate with openvswitch database(to recreate port), cause openvswitch service was not able to start in time.
From /var/log/openvswitch/ovs-vswitchd.log file:
2016-02-10T07:49:11.254Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log
2016-02-10T07:49:11.258Z|00002|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting...
2016-02-10T07:49:11.258Z|00003|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected

[Bogdan Dobrelya (bogdando)]

A tricky race condition of multiple services... Thank you for the workaround investigated!

[Bogdan Dobrelya (bogdando)]

This seems affecting generic nova-compute services as well, correct? I believe it must retry and not just fail. Adding MOS Nova

[Bogdan Dobrelya (bogdando)]

If nova-compute fails for the described simple case, the issue looks critical for Nova. Libvirt to OVS startup ordr must be addressed by system runlevels, I'm not sure if it is in the puppets scope.

[Bogdan Dobrelya (bogdando)]

Please provide logs

[Ksenia Svechnikova (kdemina)]

We have an issue with snapshot - they are not created with RC1 iso.

I've compressed some logs for nova-compute, ovs and libvirt

[Aleksander Mogylchenko (amogylchenko)]

Libvirt generally does not require OVS to work, hence there are no startup dependencies on it. This is a specific deployment scenario, and should be fixed in Puppet manifests and/or Nova init scripts.

[Roman Podoliaka (rpodolyaka)]

So we actually *do* have a dependency on libvirtd in nova-compute upstart script in 8.0:

https://review.fuel-infra.org/gitweb?p=openstack-build/nova-build.git;a=blob;f=trusty/debian/nova-compute.init.in;h=7367f19eb877ccc928a9ee0e3d8b69763a516fd8;hb=refs/heads/openstack-ci/fuel-8.0/liberty

My current understanding is that this kind of dependency between services is simply not enough in the corner case: the fact that libvirtd daemon has started *does not* mean it has bound the unix domain socket.

The same problem can *possibly* be reproduced on every single boot of a compute node. Although, we haven't seen it yet on our CI or in the field.

The bottom line is, it's possible to run into this problem in the corner case, but it must be really rare, otherwise we would see it much more often by now. Thus, I suggest we downgrade the importance from Critical to High or even further to Medium.

[Roman Podoliaka (rpodolyaka)]

And there is an easy work around for this - restart the nova-compute service.

[Roman Podoliaka (rpodolyaka)]

How should nova-compute behave when it failed to connect to the libvirtd socket?

There is always a trade-off between failing fast and retrying. Currently, the implemented way is to fail fast, thus, you see that the affected service automatically disables itself. So this is expected behavior.

If we were to fix this, I'd strongly suggest doing that by the means of systemd (i.e. bind a socket in systemd) rather than adding retries to nova-compute.

[Roman Podoliaka (rpodolyaka)]

To clarify my point:

Disabling the service when libvirtd is not available is the right thing to do, as otherwise it'd be possible to e.g. schedule a new VM on this particular compute node, which would eventually fail.

Whether or not nova-compute should try to reconnect to libvirtd when it becomes available is a topic for another discussion, IMO. Currently it's up to operator/monitoring to do that. We could potentially improve here and do that automatically, but that does not sound critical to me for 8.0.

[slava valyavskiy (slava-val-al)]

We have it constantly reproduced...at least I was able to reproduce it several time in a row.

[Roman Podoliaka (rpodolyaka)]

Ok, so after taking a closer a look, I can see the following.

We use an upstart script for nova-compute and an init.d script for libvirtd (because we use a debian package...). The former *does not* and *can't* depend on the init.d service start, unless the latter is modified to send an explicit event to upstart by the means of initctl or we provide an upstart script for libvirtd.

Curiously, we have never run into this before. The only reason Ksenia and Slava ran into this is because in their environment upstart was stuck on ntp when running rc2 init.d scripts (ntp is 23rd, while libvirtd is 28th):

http://paste.openstack.org/show/486569/

and upstart starts nova-compute in parallel with rc2 scripts.

[Roman Podoliaka (rpodolyaka)]

We have to fix this properly by ensuring the dependency between nova-compute and libvirtd, still this should probably go to a MU1 as it works "most of the time" and there is a simple workaround (restart nova-compute), so technically this is High, not Critical.

[Timur Nurlygayanov (tnurlygayanov)]

Moved to 8.0-updates, we are in HCF for 8.0

[Artem Panchenko (apanchenko-8)]

Got the same issue on baremetal lab, 2 of 3 computes were offline after cluster reboot. Libvirt daemon was started after Nova compute:

root@node-6:~# grep -E -i 'libvirt|compute' /var/log/boot.log
 * Starting OpenStack Compute [ OK ]
 * Starting libvirt management daemon libvirtd [ OK ]

root@node-6:~# ps aux | grep libvirt
root 4689 0.0 0.0 748964 10688 ? Sl 18:35 0:00 /usr/sbin/libvirtd -d -d -l
root 44068 0.0 0.0 10460 940 pts/0 R+ 22:38 0:00 grep --color=auto libvirt
root@node-6:~# ps aux | grep nova
nova 3968 0.0 0.7 1919088 116496 ? Ssl 18:35 0:03 /usr/bin/python /usr/bin/nova-compute --config-file=/etc/nova/nova.conf --config-file=/etc/nova/nova-compute.conf --log-file=/var/log/nova/nova-compute.log
root 44129 0.0 0.0 10464 932 pts/0 S+ 22:39 0:00 grep --color=auto nova

root@node-6:~# stat /proc/4689/cmdline
  File: ‘/proc/4689/cmdline’
  Size: 0 Blocks: 0 IO Block: 1024 regular empty file
Device: 3h/3d Inode: 21603 Links: 1
Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2016-02-10 18:35:32.997506503 +0000
Modify: 2016-02-10 18:35:32.997506503 +0000
Change: 2016-02-10 18:35:32.997506503 +0000
 Birth: -
root@node-6:~# stat /proc/3968/cmdline
  File: ‘/proc/3968/cmdline’
  Size: 0 Blocks: 0 IO Block: 1024 regular empty file
Device: 3h/3d Inode: 19716 Links: 1
Access: (0444/-r--r--r--) Uid: ( 107/ nova) Gid: ( 114/ nova)
Access: 2016-02-10 18:35:26.158112335 +0000
Modify: 2016-02-10 18:35:26.158112335 +0000
Change: 2016-02-10 18:35:26.158112335 +0000
 Birth: -

I also think that the bug is caused by missconfiguration of 'libvirt-daemon-system' package, currently 'libritd' daemon is not managed by upstart:

root@node-6:~# initctl list | grep -i -E 'libvirt|compute'
nova-compute start/running, process 3968

root@node-6:~# service --status-all 2>&1| grep -i -E 'libvirt|compute'
 [ + ] libvirt-guests
 [ + ] libvirtd
 [ + ] nova-compute

So 'Should-Start' with 'libvirt' boot_facility in Nova compute init script makes no sense now.

[Roman Podoliaka (rpodolyaka)]

Artem, correct.

But it turned out we don't actually use the init script of nova-compute at all - we have an upstart one for it (which does not specify any dependencies...).

So the problem is with these pacakges from Debian - libvirtd and nova-compute, which are not properly integrated with Ubuntu Trusty.

I suggest we add these to release notes in 8.0, and do the following in 8.0-mu1/9.0:

- add an upstart script for libvirtd

- add a dependency on libvirtd to nova-compute upstart script

- try to re-connect to libvirtd from nova-compute (9.0)

[Roman Podoliaka (rpodolyaka)]

Alexander M., please take a look at this. I suggest we proceed with the plan from the comment above ^

Let's start with adding an upstart script for libvirtd.

[Ksenia Svechnikova (kdemina)]

Also affect test with reboot of compute/fuel master pair:

Scenario:
1. Deploy cluster with two computes and three controllers
2. Migrate fuel master to compute
3. Reboot compute with fuel-master via reboot -f/reboot command
4. Wait till compute and fuel-master come up
5. Reconnect to fuel-master
6. Run OSTF tests
7. Run Network check
8. Check statuses for master’s services

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/16929

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Roman Podoliaka <email address hidden>
Review: https://review.fuel-infra.org/16987

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/16987

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/17946

[Aleksander Mogylchenko (amogylchenko)]

maintenance team, please check CR to 8.0 and consider merging it. The same patch will go to master shortly:
https://review.fuel-infra.org/#/c/17946/

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Bozhenko (alexbozhenko)]

@rpodolyaka:
Roman, even if you was confused with this:

>>So we actually *do* have a dependency on libvirtd in nova-compute upstart script in 8.0:

Should we finally remove all SysVinit scripts for those services, which are using upstart from our packages?
List can be obtained using this script:
for p in $(dpkg -l | awk '/mos/ {print $2}'); do filelist=$(dpkg -L $p); upstrt_cnt=$(echo $filelist | grep '/etc/init/' | wc -l); sysv_cnt=$(echo $filelist $p | grep '/etc/init.d/'| wc -l); if [[ $upstrt_cnt -gt 0 && $sysv_cnt -gt 0 ]]; then echo "$p: $upstrt_cnt $sysv_cnt"; fi; done

And move all possible services to upstart?

At least on compute nodes these services are:
root@node-2:/etc/rc2.d# ll S20openvswitch-switch
lrwxrwxrwx 1 root root 28 Mar 16 14:35 S20openvswitch-switch -> ../init.d/openvswitch-switch*
root@node-2:/etc/rc2.d# ll S28libvirtd
lrwxrwxrwx 1 root root 18 Mar 16 14:38 S28libvirtd -> ../init.d/libvirtd*
root@node-2:/etc/rc2.d# ll S29libvirt-guests
lrwxrwxrwx 1 root root 24 Mar 16 14:38 S29libvirt-guests -> ../init.d/libvirt-guests*

It is also related to these bugs:
https://bugs.launchpad.net/mos/+bug/1507069
https://bugs.launchpad.net/fuel/+bug/1552017
https://bugs.launchpad.net/mos/+bug/1540648
https://bugs.launchpad.net/mos/+bug/1393512
https://bugs.launchpad.net/ubuntu/+source/nova/+bug/907152

[Alexander Bozhenko (alexbozhenko)]

Also, regarding proposal in #20:

>- add an upstart script for libvirtd
This is better to do anyway.

>- add a dependency on libvirtd to nova-compute upstart script
How we can add dependency, if nova-compute may not use libvirt, but different hypervisor instead?

>- try to re-connect to libvirtd from nova-compute (9.0)
Should we probably force nova-compute to die if it is not connected to libvirt, to allow upstart's respawn stanza to work?
Note that even with error
 libvirtError: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory
in /var/log/nova/nova-compute.log
service continues to run, and it doesn't let respawn stanza to work

It is customer-found now, we have a ticket from customer facing this issue.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/17946
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: 7b99f649800793795af04f928de5d710b3e68b2a
Author: Aleksandr Mogylchenko <email address hidden>
Date: Wed Apr 27 10:50:43 2016

Switch libvirtd to upstart

Closes-Bug: #1543951
Change-Id: I05f6eb7b7eb5b10235aa55dffe0940ba2195e888

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Aleksandr Mogylchenko <email address hidden>
Review: https://review.fuel-infra.org/20176

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20176
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: b109ae4033b0eb02adfee7c55d163081975e7551
Author: Aleksandr Mogylchenko <email address hidden>
Date: Wed Apr 27 16:11:37 2016

Switch libvirtd to upstart

Closes-Bug: #1543951
Change-Id: I05f6eb7b7eb5b10235aa55dffe0940ba2195e888
(cherry picked from commit 7b99f649800793795af04f928de5d710b3e68b2a)

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20345
Submitter: Igor Yozhikov <email address hidden>
Branch: master

Commit: eb63fba049c18cd2ea860b8c9ef05c04c0f10dd5
Author: Dmitry Teselkin <email address hidden>
Date: Fri May 13 10:34:41 2016

Fix reading libvirt default config on start

On debian systems /etc/default/libvirtd should be used
for setting daemon configuration options. The important
option is -l [listen]. The second config file
/etc/libvirt/libvirtd.conf is used as well.
The root cause of the LP#1576688 was missing of reading
config parameters from config files when migrated to the
Upstart

While at it make sure /var/run/libvirt directory exists
before starting libvirtd (the content of /var/run is not
preserved across reboots) so the daemon can start after
rebooting the host.

Change-Id: I1cd150a9285244d19d64e44737caf89d38703f23
Closes-bug: #1576688
Closes-bug: #1543951

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Dmitry Teselkin <email address hidden>
Review: https://review.fuel-infra.org/20674

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/20674
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 5639d6ba7d73febec521ca8548edb8670d4c6b8f
Author: Dmitry Teselkin <email address hidden>
Date: Mon May 16 07:44:07 2016

Fix reading libvirt default config on start

On debian systems /etc/default/libvirtd should be used
for setting daemon configuration options. The important
option is -l [listen]. The second config file
/etc/libvirt/libvirtd.conf is used as well.
The root cause of the LP#1576688 was missing of reading
config parameters from config files when migrated to the
Upstart

While at it make sure /var/run/libvirt directory exists
before starting libvirtd (the content of /var/run is not
preserved across reboots) so the daemon can start after
rebooting the host.

Change-Id: I1cd150a9285244d19d64e44737caf89d38703f23
Closes-bug: #1576688
Closes-bug: #1543951
(cherry picked from commit eb63fba049c18cd2ea860b8c9ef05c04c0f10dd5)

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 8.0
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/20853

[Roman Podoliaka (rpodolyaka)]

We also need to add a dependency on libvirtd to nova-compute-(qemu|kvm) upstart script (note, that we don't need this if e.g. vmware, ironic or any other virt driver used) to close this properly (although, in 9.0 it already works due to the fact that nova-compute will die and be restarted by upstart, if libvirtd is not available).

Ivan, could you please help with this?

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 9.0/mitaka
Change author: Ivan Udovichenko <email address hidden>
Review: https://review.fuel-infra.org/20995

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ivan Udovichenko <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/20995
Reason: Replaced with: https://review.fuel-infra.org/#/c/21232/

[Dmitry Sutyagin (dsutyagin)]

I've hit this issue on my 8.0 test environment (no MUs installed) - wanted to file a bug but it's already present. Just sharing.

[Denis Meltsaykin (dmeltsaykin)]

Dmitry, yes, this bug reproduces 100% of reboots. We are working on fixing it in libvirt & nova-compute-(qemu|kvm). Hopefully, it will be included into MU2.

[Ivan Udovichenko (iudovichenko)]

Fix was proposed in CR-21232: https://review.fuel-infra.org/#/c/21232/

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ivan Udovichenko <email address hidden>
Review: https://review.fuel-infra.org/21576

[Ksenia Svechnikova (kdemina)]

@Sergey
When you add swarm-fail tag and raise priority, please add any information about what TC is failed by this issue. As for now it's not clear what Milestone it affects and at what test should we look

[Sergey Shevorakov (sshevorakov)]

Here is the case: https://mirantis.testrail.com/index.php?/tests/view/6630417

[Sergey Shevorakov (sshevorakov)]

I was talking about 9.0, since the bug appeared in 9.0 ISO #427 Swarm results.
https://mirantis.testrail.com/index.php?/plans/view/12374

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21576
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 117007286cd3ca6b5a8e7a6fcfbcf4f129af6fc7
Author: Ivan Udovichenko <email address hidden>
Date: Mon Jun 6 10:50:52 2016

[MOS 8.0] Update nova package

If nova-compute is being used in conjunction with KVM or QEMU, nova-compute
service should be started after libvirtd service.

(cherry-picked from b57b9a3f456f40a1ae3470c9ac85f81101068b5d)

Related-Bug: #1543951
Change-Id: Ib26fd39c82644e88ca7c785e74505e8cc86b0be7

[Ksenia Svechnikova (kdemina)]

Verified on 9.0-mos-490 ISO

Compute service is running after reboot of compute, virt node

[Alexander Bozhenko (alexbozhenko)]

@Ksenia, could you please verify on usual setup(without reduced footprint) too. Because we saw same errors on ususal setup(3 controllers) as well.

[Ksenia Svechnikova (kdemina)]

On RC2, mos-9.0 I've checked hard reboot of compute node on cluster (3 controllers + 2 computes). After it's up, nova service-list show that the services on that compute are enabled

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU2 updates.

Actual results:
If reboot compute node-4 on cluster (3 controllers + 3 computes) and after node is up, nova-compute service is enabled on this node, there is no errors in nova-compute.log.
root@node-1:~# nova service-list | grep nova-compute
| 34 | nova-compute | node-7.test.domain.local | nova | enabled | up | 2016-06-29T05:51:21.000000
| 37 | nova-compute | node-4.test.domain.local | nova | enabled | up | 2016-06-29T05:50:49.000000
| 40 | nova-compute | node-6.test.domain.local | nova | enabled | up | 2016-06-29T05:50:33.000000

[Sergey Novikov (snovikov)]

Verified on MOS 10.0 (RC #2)