Neutron L3 agent state directory is not persistent

[Adam Oswick]

What happened:

Whenever a new Neutron container is deployed, we sometimes see errors such as "pyroute2.netlink.exceptions.NetlinkError: (101, 'Network is unreachable')".

It looks like this is due to the file '/var/lib/neutron/fip-linklocal-networks' not being persisted between container upgrades.

What is expected:

Based on https://opendev.org/openstack/neutron/src/commit/09a6d126ac306b2e8564732562c87d3831450f90/neutron/agent/l3/link_local_allocator.py#L35-L38, it seems that Neutron expects at least some files in the state_path (/var/lib/neutron by default) to be persistent across restarts.

Therefore, it seems like it may be a good option to have another volume in the default volumes for the Neutron L3 agent to provide this persistence.

How to reproduce:
- Deploy OpenStack with Neutron DVR enabled
- Spawn VMs on hypervisors (so that routers, namespaces, etc. are setup)
- Upgrade the Neutron container
- Search for pyroute2.netlink.exceptions.NetlinkError exceptions in the Neutron L3 agent logs

Environment
- OS: Almalinux 8 & 9
- Kolla-Ansible version: Zed
- Docker image distro: Rocky Linux 9.1
- Docker images: self built
- Neutron DVR enabled (the above pyroute2 issue is likely only applicable when DVR is in use)

[Adam Oswick (adamoswick)]

Created https://review.opendev.org/c/openstack/kolla-ansible/+/877036 to implement this

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/877036
Committed: https://opendev.org/openstack/kolla-ansible/commit/25244517e162a02e73503c56f9f370f9a71a1f7d
Submitter: "Zuul (22348)"
Branch: master

commit 25244517e162a02e73503c56f9f370f9a71a1f7d
Author: Adam Oswick <email address hidden>
Date: Tue Sep 26 11:40:52 2023 +0100

    Persist Neutron agent state files in volume

    The Neutron L3 agent stores state at state_path (/var/lib/neutron by
    default) and it is expected that these files persist across restarts.

    This change updates the Neutron state_path value to
    /var/lib/neutron/kolla (which is where the neutron_metadata_socket
    volume is mounted) so that these state files are stored there.

    Change-Id: I739aaf9e2d0b2b2e7f7b8f60ef8c2111d6873cef
    Signed-off-by: Adam Oswick <email address hidden>
    Closes-Bug: #2009884

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904512

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904513

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/904515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904514
Committed: https://opendev.org/openstack/kolla-ansible/commit/1eae44d10a436a978afd89849b6c28d71bf3a759
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 1eae44d10a436a978afd89849b6c28d71bf3a759
Author: Adam Oswick <email address hidden>
Date: Tue Sep 26 11:40:52 2023 +0100

    Persist Neutron agent state files in volume

    The Neutron L3 agent stores state at state_path (/var/lib/neutron by
    default) and it is expected that these files persist across restarts.

    This change updates the Neutron state_path value to
    /var/lib/neutron/kolla (which is where the neutron_metadata_socket
    volume is mounted) so that these state files are stored there.

    Change-Id: I739aaf9e2d0b2b2e7f7b8f60ef8c2111d6873cef
    Signed-off-by: Adam Oswick <email address hidden>
    Closes-Bug: #2009884
    (cherry picked from commit 25244517e162a02e73503c56f9f370f9a71a1f7d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904512
Committed: https://opendev.org/openstack/kolla-ansible/commit/bf9ed407316eb335beea30bd5a35dc0e6b4cddb3
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit bf9ed407316eb335beea30bd5a35dc0e6b4cddb3
Author: Adam Oswick <email address hidden>
Date: Tue Sep 26 11:40:52 2023 +0100

    Persist Neutron agent state files in volume

    The Neutron L3 agent stores state at state_path (/var/lib/neutron by
    default) and it is expected that these files persist across restarts.

    This change updates the Neutron state_path value to
    /var/lib/neutron/kolla (which is where the neutron_metadata_socket
    volume is mounted) so that these state files are stored there.

    Change-Id: I739aaf9e2d0b2b2e7f7b8f60ef8c2111d6873cef
    Signed-off-by: Adam Oswick <email address hidden>
    Closes-Bug: #2009884
    (cherry picked from commit 25244517e162a02e73503c56f9f370f9a71a1f7d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904513
Committed: https://opendev.org/openstack/kolla-ansible/commit/3ce2fa84f25a07f16d74405e162246009518cd3b
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 3ce2fa84f25a07f16d74405e162246009518cd3b
Author: Adam Oswick <email address hidden>
Date: Tue Sep 26 11:40:52 2023 +0100

    Persist Neutron agent state files in volume

    The Neutron L3 agent stores state at state_path (/var/lib/neutron by
    default) and it is expected that these files persist across restarts.

    This change updates the Neutron state_path value to
    /var/lib/neutron/kolla (which is where the neutron_metadata_socket
    volume is mounted) so that these state files are stored there.

    Change-Id: I739aaf9e2d0b2b2e7f7b8f60ef8c2111d6873cef
    Signed-off-by: Adam Oswick <email address hidden>
    Closes-Bug: #2009884
    (cherry picked from commit 25244517e162a02e73503c56f9f370f9a71a1f7d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/904515
Committed: https://opendev.org/openstack/kolla-ansible/commit/3e2f6b85671b062953f24ee43d3fdd4f9cdb4aa4
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 3e2f6b85671b062953f24ee43d3fdd4f9cdb4aa4
Author: Adam Oswick <email address hidden>
Date: Tue Sep 26 11:40:52 2023 +0100

    Persist Neutron agent state files in volume

    The Neutron L3 agent stores state at state_path (/var/lib/neutron by
    default) and it is expected that these files persist across restarts.

    This change updates the Neutron state_path value to
    /var/lib/neutron/kolla (which is where the neutron_metadata_socket
    volume is mounted) so that these state files are stored there.

    Change-Id: I739aaf9e2d0b2b2e7f7b8f60ef8c2111d6873cef
    Signed-off-by: Adam Oswick <email address hidden>
    Closes-Bug: #2009884
    (cherry picked from commit 25244517e162a02e73503c56f9f370f9a71a1f7d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 16.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 17.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.11.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 15.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 18.0.0.0rc1 release candidate.