nova-compute doesn't start due to libvirtd --listen restriction

[Michael]

hello,

after upgrading from Openstack Train to Ussuri, running on Ubuntu 18.04, the nova-compute service doesn't start due to the libvirtd service not starting because of the --listen option being enabled in the /etc/default/libvirtd file and so, the nova-compute service fail to reach the socket that doesn't exist.

i have found other bug report and patch that claim to have fixed this issue but it's not, at least not on the package installed version of openstack on ubuntu 18.04 (https://bugs.launchpad.net/puppet-nova/+bug/1880619)

is nova-compute able to use another mechanism to connect to libvirtd ? and thus not needing the --listen option ?

[Takashi Kajinami (kajinamit)]

I think this is not an issue with nova but the one with puppet-nova.

Actually we fixed the configration related to libvirt on CentOS while we fixed that mentioned bug, but we still need the same fix for Ubuntu which has libvirt 6.0 recently.

We already have a proposed fix for the issue https://review.opendev.org/#/c/749603 so I think we can move this forward to address the issue.

[Michael (michaelh01)]

yes it does seem to be the same issue, but i'am not using puppet-nova. i found a workaround, starting the daemon with /usr/sbin/libvirtd -d --listen but i can't seem to find your proposed fix for the non puppet install ? would it be changes in the config files ?

Thanks !

[Takashi Kajinami (kajinamit)]

If you run recent libvirtd under systemd, you should use appropriate .socket instead of --listen option.
Try:
 - Remove --listen from LIBVIRTD_ARGS in /etc/sysconfig/libvirtd
 - Enable libvirtd-tcp.socket or libvirtd-tls.socket in systemd

See the following patch in puppet-nova for your reference.
https://review.opendev.org/#/c/730689/11/manifests/migration/libvirt.pp

The socket you use depends on the protocol in migration.

[Balazs Gibizer (balazs-gibizer)]

This is not a nova issue. I it is a libvirtd configuration issue. Nova needs to connect to libvirtd via a socket. I'm marking the nova side of this bug Invalid.

[Takashi Kajinami (kajinamit)]

Yeah I agree that this is not a bug but a kind of configuration issue in libvirt.

Sorry I mentioned wrong path and variable used in CentOS, but you should look into the file located at /etc/defaults/libvirtd and update the line including libvirtd_opts, If I understand the implementation for Ubuntu correctly.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/756773

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/749603
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=a44641df76730484ec82d119c9473251bee7ad72
Submitter: Zuul
Branch: master

commit a44641df76730484ec82d119c9473251bee7ad72
Author: Tobias Urdin <email address hidden>
Date: Thu Sep 3 01:19:05 2020 +0200

    Ubuntu has libvirt 6.0

    The UCA repo has libvirt 6.0 from Ussuri
    on Bionic and later.

    Closes-Bug: #1898553
    Change-Id: I8146f5b073e812f3cac472fc5e27976f17d248b4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/756773
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=20585762919c49e6dffab58664949b23d2306eae
Submitter: Zuul
Branch: master

commit 20585762919c49e6dffab58664949b23d2306eae
Author: Takashi Kajinami <email address hidden>
Date: Thu Oct 8 23:33:49 2020 +0900

    Cleanup listen option from libvirtd service file

    This patch ensures that listen option is removed from service file when
    libvirt>=5.6 is used, since the option might be left if the deployment
    is updated or upgraded from old version.

    Related-Bug: #1898553
    Related-Bug: #1880619
    Change-Id: Ifb0f5e1d7db31885636f689166ff666b049e9414

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/757897

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/758018

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/758019

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/758024

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/757897
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=0fb9b479c8261afaf895e9ecb611d6515a1e4de7
Submitter: Zuul
Branch: stable/victoria

commit 0fb9b479c8261afaf895e9ecb611d6515a1e4de7
Author: Tobias Urdin <email address hidden>
Date: Thu Sep 3 01:19:05 2020 +0200

    Ubuntu has libvirt 6.0

    The UCA repo has libvirt 6.0 from Ussuri
    on Bionic and later.

    Closes-Bug: #1898553
    Change-Id: I8146f5b073e812f3cac472fc5e27976f17d248b4
    (cherry picked from commit a44641df76730484ec82d119c9473251bee7ad72)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758018
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=8ab5642322512cd59d2959dbcfe968d5fb4db86d
Submitter: Zuul
Branch: stable/victoria

commit 8ab5642322512cd59d2959dbcfe968d5fb4db86d
Author: Takashi Kajinami <email address hidden>
Date: Thu Oct 8 23:33:49 2020 +0900

    Cleanup listen option from libvirtd service file

    This patch ensures that listen option is removed from service file when
    libvirt>=5.6 is used, since the option might be left if the deployment
    is updated or upgraded from old version.

    Related-Bug: #1898553
    Related-Bug: #1880619
    Change-Id: Ifb0f5e1d7db31885636f689166ff666b049e9414
    (cherry picked from commit 20585762919c49e6dffab58664949b23d2306eae)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/752788
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=6fed1eff23759e609b59fd998eeec528df6e3d07
Submitter: Zuul
Branch: stable/ussuri

commit 6fed1eff23759e609b59fd998eeec528df6e3d07
Author: Tobias Urdin <email address hidden>
Date: Thu Sep 3 01:19:05 2020 +0200

    Ubuntu has libvirt 6.0

    The UCA repo has libvirt 6.0 from Ussuri
    on Bionic and later.

    Closes-Bug: #1898553
    Change-Id: I8146f5b073e812f3cac472fc5e27976f17d248b4
    (cherry picked from commit a44641df76730484ec82d119c9473251bee7ad72)
    (cherry picked from commit 2d52ce1639885d66317424377cad4d166bb4f85f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758019
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=6dafd844a0b6a82d254e7dffaec426779a9a995a
Submitter: Zuul
Branch: stable/ussuri

commit 6dafd844a0b6a82d254e7dffaec426779a9a995a
Author: Takashi Kajinami <email address hidden>
Date: Thu Oct 8 23:33:49 2020 +0900

    Cleanup listen option from libvirtd service file

    This patch ensures that listen option is removed from service file when
    libvirt>=5.6 is used, since the option might be left if the deployment
    is updated or upgraded from old version.

    Related-Bug: #1898553
    Related-Bug: #1880619
    Change-Id: Ifb0f5e1d7db31885636f689166ff666b049e9414
    (cherry picked from commit 20585762919c49e6dffab58664949b23d2306eae)
    (cherry picked from commit 386a5d00285155cc3bc7a5f190fb6c86f05bff23)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 18.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 17.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-nova 16.5.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Takashi Kajinami <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/puppet-nova/+/756691
Reason: This is duplicate of https://review.opendev.org/c/openstack/puppet-nova/+/749603