kolla-ansible

Kolla fails to deploy neutron when ipv6 is disabled

Series yoga
Bug #1906306

Bug #1906306 reported by Imtiaz Chowdhury on 2020-11-30

This bug affects 4 people

	Status	Importance	Assigned to
kolla-ansible	Fix Released	Medium	Roman Krček
Antelope	Fix Committed	Medium	Unassigned
Bobcat	Fix Committed	Medium	Unassigned
Caracal	Fix Released	Medium	Roman Krček
Yoga	Fix Committed	Undecided	Unassigned
Zed	Fix Released	Medium	Unassigned

Bug Description

What happened:

While trying to deploy an all-in-one instance using Kolla-ansible, the Ansible scripts failed with the following error message:

TASK [neutron : Setting sysctl values] **************************************************************************************************************************************************
ok: [10.140.145.117] => (item={'name': 'net.ipv4.ip_forward', 'value': 1}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.ip_forward", "value": 1}}
ok: [10.140.145.117] => (item={'name': 'net.ipv4.conf.all.rp_filter', 'value': 0}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.conf.all.rp_filter", "value": 0}}
ok: [10.140.145.117] => (item={'name': 'net.ipv4.conf.default.rp_filter', 'value': 0}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.conf.default.rp_filter", "value": 0}}
ok: [10.140.145.117] => (item={'name': 'net.ipv4.neigh.default.gc_thresh1', 'value': 128}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.neigh.default.gc_thresh1", "value": 128}}
ok: [10.140.145.117] => (item={'name': 'net.ipv4.neigh.default.gc_thresh2', 'value': 28672}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.neigh.default.gc_thresh2", "value": 28672}}
ok: [10.140.145.117] => (item={'name': 'net.ipv4.neigh.default.gc_thresh3', 'value': 32768}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv4.neigh.default.gc_thresh3", "value": 32768}}
failed: [10.140.145.117] (item={'name': 'net.ipv6.neigh.default.gc_thresh1', 'value': '128'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv6.neigh.default.gc_thresh1", "value": "128"}, "msg": "Failed to reload sysctl: net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.default.rp_filter = 0\nnet.ipv4.ip_forward = 1\nnet.ipv4.neigh.default.gc_thresh1 = 128\nnet.ipv4.neigh.default.gc_thresh2 = 28672\nnet.ipv4.neigh.default.gc_thresh3 = 32768\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory\n"}
failed: [10.140.145.117] (item={'name': 'net.ipv6.neigh.default.gc_thresh2', 'value': '28672'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv6.neigh.default.gc_thresh2", "value": "28672"}, "msg": "Failed to reload sysctl: net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.default.rp_filter = 0\nnet.ipv4.ip_forward = 1\nnet.ipv4.neigh.default.gc_thresh1 = 128\nnet.ipv4.neigh.default.gc_thresh2 = 28672\nnet.ipv4.neigh.default.gc_thresh3 = 32768\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory\n"}
failed: [10.140.145.117] (item={'name': 'net.ipv6.neigh.default.gc_thresh3', 'value': '32768'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "net.ipv6.neigh.default.gc_thresh3", "value": "32768"}, "msg": "Failed to reload sysctl: net.bridge.bridge-nf-call-iptables = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.conf.all.rp_filter = 0\nnet.ipv4.conf.default.rp_filter = 0\nnet.ipv4.ip_forward = 1\nnet.ipv4.neigh.default.gc_thresh1 = 128\nnet.ipv4.neigh.default.gc_thresh2 = 28672\nnet.ipv4.neigh.default.gc_thresh3 = 32768\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh1: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh2: No such file or directory\nsysctl: cannot stat /proc/sys/net/ipv6/neigh/default/gc_thresh3: No such file or directory\n"}

What is expected:
The error seems to be caused when ipv6 is disabled on a host (defined in /boot/grub2/grub.cfg). When ipv6 is disabled, sysctl cannot set properties for values for e.g. ipv6/neigh/default/gc_thresh1.

Environment:

OS: CentOS Linux release 8.2.2004 (Core)
Kernel: 4.18.0-193.28.1.el8_2.x86_64
Kolla-Ansible version: 10.1.0

Mark Goddard (mgoddard) on 2020-12-08

Changed in kolla-ansible:
status:	New → Triaged
importance:	Undecided → High
importance:	High → Medium

OpenStack Infra (hudson-openstack) on 2024-01-25

Changed in kolla-ansible:
status:	Triaged → In Progress

Revision history for this message

Maksim Malchuk (mmalchuk) wrote on 2024-02-10:

Fix proposed to the branch master: https://review.opendev.org/c/openstack/kolla-ansible/+/905831

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-10: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/912351

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-22: Fix merged to kolla-ansible (master)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/905831
Committed: https://opendev.org/openstack/kolla-ansible/commit/9301e82d7bd56e214e406319ee75db42b03a5167
Submitter: "Zuul (22348)"
Branch: master

commit 9301e82d7bd56e214e406319ee75db42b03a5167
Author: Roman Krček <email address hidden>
Date: Mon Jan 15 15:19:42 2024 +0100

Add conditionals for IPv6 sysctl settings

This way the playbooks won't try to set ipv6 systemctl options
unless ipv6 is available on the system.

Closes-bug: #1906306
Change-Id: Icccfc1c509179c3cfd59650b7917a637f9af9646

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-22: Fix proposed to kolla-ansible (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/913818

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-22: Fix proposed to kolla-ansible (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/913819

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-22: Fix proposed to kolla-ansible (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/913820

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-28: Fix merged to kolla-ansible (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/913818
Committed: https://opendev.org/openstack/kolla-ansible/commit/de5af72b6345dbefb53c92917f824997a7ad3545
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit de5af72b6345dbefb53c92917f824997a7ad3545
Author: Roman Krček <email address hidden>
Date: Mon Jan 15 15:19:42 2024 +0100

Add conditionals for IPv6 sysctl settings

This way the playbooks won't try to set ipv6 systemctl options
unless ipv6 is available on the system.

    Closes-bug: #1906306
    Change-Id: Icccfc1c509179c3cfd59650b7917a637f9af9646
    (cherry picked from commit 9301e82d7bd56e214e406319ee75db42b03a5167)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-28: Fix merged to kolla-ansible (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/913819
Committed: https://opendev.org/openstack/kolla-ansible/commit/f56396da365955fbfcf3fbd2ba03acd689dbb306
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit f56396da365955fbfcf3fbd2ba03acd689dbb306
Author: Roman Krček <email address hidden>
Date: Mon Jan 15 15:19:42 2024 +0100

Add conditionals for IPv6 sysctl settings

This way the playbooks won't try to set ipv6 systemctl options
unless ipv6 is available on the system.

    Closes-bug: #1906306
    Change-Id: Icccfc1c509179c3cfd59650b7917a637f9af9646
    (cherry picked from commit 9301e82d7bd56e214e406319ee75db42b03a5167)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-03-28: Fix merged to kolla-ansible (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/913820
Committed: https://opendev.org/openstack/kolla-ansible/commit/6587d45c0458da3d2e813800236d46808bb07edf
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 6587d45c0458da3d2e813800236d46808bb07edf
Author: Roman Krček <email address hidden>
Date: Mon Jan 15 15:19:42 2024 +0100

Add conditionals for IPv6 sysctl settings

This way the playbooks won't try to set ipv6 systemctl options
unless ipv6 is available on the system.

    Closes-bug: #1906306
    Change-Id: Icccfc1c509179c3cfd59650b7917a637f9af9646
    (cherry picked from commit 9301e82d7bd56e214e406319ee75db42b03a5167)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-04-09: Fix proposed to kolla-ansible (unmaintained/yoga)

#10

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/915156

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-04-24: Related fix merged to kolla-ansible (master)

#11

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/912351
Committed: https://opendev.org/openstack/kolla-ansible/commit/e2a0d1f59bc9a8d3a90c0144887dd5ad33806151
Submitter: "Zuul (22348)"
Branch: master

commit e2a0d1f59bc9a8d3a90c0144887dd5ad33806151
Author: Roman Krček <email address hidden>
Date: Sun Mar 10 21:54:41 2024 +0100

Add sysctl role

This new role will handle setting sysctl values.

It also handles cases when IPv6 setting is changed, but IPv6 is
not enabled on the system by skipping those settings.

This is an augmentation of previous patch:
Icccfc1c509179c3cfd59650b7917a637f9af9646

    Related-bug: #1906306
    Change-Id: I5d6cda3307b3d2f27c1b2995f28772523b203fe7
    Signed-off-by: Roman Krček <email address hidden>

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-04-29: Fix merged to kolla-ansible (unmaintained/yoga)

#12

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/915156
Committed: https://opendev.org/openstack/kolla-ansible/commit/e27c4159b4aefd91e842eb0e8bd8805757dac49e
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit e27c4159b4aefd91e842eb0e8bd8805757dac49e
Author: Roman Krček <email address hidden>
Date: Mon Jan 15 15:19:42 2024 +0100

Add conditionals for IPv6 sysctl settings

This way the playbooks won't try to set ipv6 systemctl options
unless ipv6 is available on the system.

    Closes-bug: #1906306
    Change-Id: Icccfc1c509179c3cfd59650b7917a637f9af9646
    (cherry picked from commit 9301e82d7bd56e214e406319ee75db42b03a5167)