neutron.provisioningblocks WSREP: referenced FK check fail

[Maximilian Stinsky]

Greetings,

I am seeing the following error in our galera cluster a couple of times a day:

[ERROR] InnoDB: WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron`.`provisioningblocks`

This only happens for neutron and always for the provisioningblocks table.
Sadly I am not 100% sure when this error happened the first time or with what upgrade it was introduced.

I am seeing this error in our production cluster with neutron ussuri aswell as our lab cluster which is neutron victoria.

Our galera cluster is running with mariadb 10.5.8.

Is this a known error or are there any hints on what to do about it?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Maximilian:

I saw this issue reported in other occasions [1][2]. I could be an issue in the DB engine. However, checking the code (before Zed release), I think we can improve the only method that is creating the "provisioningblocks" registers [3]. In this method we first retrieve the "standard_attr_id" and use it to create the "provisioningblocks". But this method can be called outside a DB context. That means the "standard_attr" register can be deleted before the "provisioningblocks" is created.

Adding a writer context to the method (as in Zed release), we can ensure the "standard_attr_id" FK exists when the register is created. That will be guaranteed by the transactionality of the DB engine.

Regards.

[1]https://jira.mariadb.org/browse/MDEV-18562
[2]https://meetings.opendev.org/irclogs/%23openstack-kolla/%23openstack-kolla.2020-07-29.log.html
[3]https://github.com/openstack/neutron/blob/d847b52c4f36e17c9c360320d160a0d05330a71c/neutron/db/provisioning_blocks.py#L107-L143

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/859865

[Maximilian Stinsky (mstinsky)]

Hi Rodolfo,

is this something I could safely backport to victoria aswell?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/860005

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/860215

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/860216

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/860217

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/860218

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860215
Committed: https://opendev.org/openstack/neutron/commit/b96886b9ea8f173afc5f9b6ba905c33d1de49508
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit b96886b9ea8f173afc5f9b6ba905c33d1de49508
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55
    (cherry picked from commit 9db730764c3b6bc394c4262e68bfcec7cea65b50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860216
Committed: https://opendev.org/openstack/neutron/commit/ebbc79c7829d881aa0840132a2bc0bace6eff72b
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit ebbc79c7829d881aa0840132a2bc0bace6eff72b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55
    (cherry picked from commit 9db730764c3b6bc394c4262e68bfcec7cea65b50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860218
Committed: https://opendev.org/openstack/neutron/commit/8e9a99b589eb91547d6c26a20419470aa3e7739b
Submitter: "Zuul (22348)"
Branch: stable/train

commit 8e9a99b589eb91547d6c26a20419470aa3e7739b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55
    (cherry picked from commit 9db730764c3b6bc394c4262e68bfcec7cea65b50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860217
Committed: https://opendev.org/openstack/neutron/commit/e1a45c82f32d343a82bbb80e5533b7917e521a4d
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit e1a45c82f32d343a82bbb80e5533b7917e521a4d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55
    (cherry picked from commit 9db730764c3b6bc394c4262e68bfcec7cea65b50)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/860005
Committed: https://opendev.org/openstack/neutron/commit/9db730764c3b6bc394c4262e68bfcec7cea65b50
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 9db730764c3b6bc394c4262e68bfcec7cea65b50
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/859865
Committed: https://opendev.org/openstack/neutron/commit/99bce9d055e08d9921ebe14164a36e8d2a47f81c
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 99bce9d055e08d9921ebe14164a36e8d2a47f81c
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 28 23:32:20 2022 +0200

    [stable-only] Add writer DB context to "add_provisioning_component"

    To ensure the "standardatributes" register is present during the
    creation of the "provisioningblocks" register, the method
    "add_provisioning_component" is wrapped with a database writer
    context. That will ensure the standard attribute ID, used as a
    foreign key in the "provisioningblocks" table, is present during
    the transaction.

    Closes-Bug: #1991222
    Change-Id: If57d822ff617c2f9b5e5cade7d4ca74065376e55
    (cherry picked from commit 9db730764c3b6bc394c4262e68bfcec7cea65b50)

[Maximilian Stinsky (mstinsky)]

@Rodolfo Alonso I backported this patch to our victoria lab and still see the error.
So that did not fix it. Do you have any other idea where this could come from?

[Ammad Ali (syedammad83)]

Hi,

I am facing similar issue in my xena and yoga deployments. I am using percona xtradb cluster 8.0 and seeing these errors on different neutron tables.

2022-04-04T15:39:17.342175+05:00 187579 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`portsecuritybindings`
2022-04-05T14:18:19.801183+05:00 275503 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`ml2_port_binding_levels`
2022-04-19T11:13:26.063954+05:00 400458 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`portsecuritybindings`
2022-04-25T13:17:08.562935+05:00 919410 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`ml2_port_bindings`
2022-04-26T14:58:36.891617+05:00 1028780 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`portsecuritybindings`
2022-05-11T22:31:12.693103+05:00 2509379 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`provisioningblocks`
2022-05-12T10:46:48.540775+05:00 2540522 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`provisioningblocks`
2022-05-12T11:15:44.698450+05:00 2540497 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`portsecuritybindings`
2022-05-12T11:18:47.129934+05:00 2543106 [ERROR] [MY-011825] [InnoDB] WSREP: referenced FK check fail: Lock wait index `PRIMARY` table `neutron_ml2`.`securitygroupportbindings`

I have once opened it on percona forum here https://forums.percona.com/t/wsrep-referenced-fk-check-fail/13958

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.6.0 release.

[Stanislav Dmitriev (sdmitriev1)]

Backport to Train doesn't fix problem either. Still see the same amount "FK check fail" in the logs

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron train-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ussuri-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron victoria-eom release.