Neutron DB is made inconsistent when plugin fails

[Pavel Smirnov]

Numerous issues identified due to the fact that there is no transactional consistency between Neutrons own Database operations and implementing plugin operations.

Specifically issue identified with the use of NSXV plugin.

Situation#1:
- Load Balancer being created
- Healthmonitor created
- Pool created
- Provisioning fails due to NSX Manager API error assigning the monitor to the pool
- Yet, details of the pool association are recorded into the Neutron Database
- Subsequent deletion of the pool will fail inside NSXV plugin due to DB inconsistency

Situation#2:
- Similar situation
- Pool creation may fail due to NSX issues
- Neutron erroneously records the new pool details despite the provisioning errors
- Deletion attempts of such pool will fail

Situation#3:
- Security group creation
- Security group fails to create firewall section in NSXV, but group is recorded in Openstack
- Deletion attempts of this group will fail

General note:
- Can the MySQL Database transaction wrap be used as a general rule to prevent inconsistent DB updates where backing plugin actions result in an exception ?

Even though NSXV is clearly identified as a culprit in this case, other plugins may easily fail in exactly the same way. End to end API call consistency is important, as partial updates may harm operations in a variety of ways.

Neutron Version: Mitaka as supplied with VMWARE Integrated Openstack 3.1
NSX Version: 6.3.0

[Jakub Libosvar (libosvar)]

Mitaka is EOL now. Are you able to reproduce the issue with the latest stable Pike version?

[Jakub Libosvar (libosvar)]

Also this bug has been reported against Neutron, which typically means ML2 reference implementation, that is part of Neutron tree. Can you please verify if this also applies against the reference implementation? Otherwise, I think this bug should be opened against NSX Neutron Plugin.

[Miguel Lavalle (minsel)]

What leads you to believe that "other plugins may easily fail in exactly the same way"?

[Pav Smirnoff (itwise-net)]

Hi Guys,

@Jakub,

Not sure Pike is available to test, but I'm trying to create a test fixture
now.
Also going to get VIO 4.1, which should be bring the level higher.
In fact you may be right to point at NSX plugin anyway as it maintains a
number of own DB tables thus layering SQL and NSX API calls.

@Miguel,

General thought can be "bogus", but the lack of transaction wrap around the
API call means an SQL statement or plugin action between can fail, raise an
exception with some of the earlier activities complete. In this case,
Openstack LBaaS tables were populated, NSX API call failed and NSX id
mapping tables were not populated. If there was an imaginary "Begin
Transaction" at the front of the API call and the "Commit" at the end with
possible "Rollback" on exception, then partially successful DB writes would
have rolled back.

Regards
Pavel

On 5 February 2018 at 20:36, Miguel Lavalle <email address hidden> wrote:

> What leads you to believe that "other plugins may easily fail in exactly
> the same way"?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746707
>
> Title:
> Neutron DB is made inconsistent when plugin fails
>
> Status in neutron:
> Incomplete
>
> Bug description:
> Numerous issues identified due to the fact that there is no
> transactional consistency between Neutrons own Database operations and
> implementing plugin operations.
>
> Specifically issue identified with the use of NSXV plugin.
>
> Situation#1:
> - Load Balancer being created
> - Healthmonitor created
> - Pool created
> - Provisioning fails due to NSX Manager API error assigning the monitor
> to the pool
> - Yet, details of the pool association are recorded into the Neutron
> Database
> - Subsequent deletion of the pool will fail inside NSXV plugin due to DB
> inconsistency
>
> Situation#2:
> - Similar situation
> - Pool creation may fail due to NSX issues
> - Neutron erroneously records the new pool details despite the
> provisioning errors
> - Deletion attempts of such pool will fail
>
> Situation#3:
> - Security group creation
> - Security group fails to create firewall section in NSXV, but group is
> recorded in Openstack
> - Deletion attempts of this group will fail
>
> General note:
> - Can the MySQL Database transaction wrap be used as a general rule to
> prevent inconsistent DB updates where backing plugin actions result in an
> exception ?
>
> Even though NSXV is clearly identified as a culprit in this case,
> other plugins may easily fail in exactly the same way. End to end API
> call consistency is important, as partial updates may harm operations
> in a variety of ways.
>
> Neutron Version: Mitaka as supplied with VMWARE Integrated Openstack 3.1
> NSX Version: 6.3.0
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1746707/+subscriptions
>

--
Regards,

Pavel Smirnov
Director,

*PAVEV Limited T/A ITERRA* <http://ie.linkedin.com/in/iterra>

Phone: [phone]
Email: <email address hidden> Web: http://www.iterra.ie/
Registered address: 58 Aylmer Park, Naas, Co Kil...

[Pav Smirnoff (itwise-net)]

@Jakub,

Not sure Pike is available to test, but I'm trying to create a test fixture now.
Also going to get VIO 4.1, which should be bring the level higher.
In fact you may be right to point at NSX plugin anyway as it maintains a number of own DB tables thus layering SQL and NSX API calls.

@Miguel,

General thought can be "bogus", but the lack of transaction wrap around the API call means an SQL statement or plugin action between can fail, raise an exception with some of the earlier activities complete. In this case, Openstack LBaaS tables were populated, NSX API call failed and NSX id mapping tables were not populated. If there was an imaginary "Begin Transaction" at the front of the API call and the "Commit" at the end with possible "Rollback" on exception, then partially successful DB writes would have rolled back.

[Jakub Libosvar (libosvar)]

We try to do rollbacks in the ML2 code in case mechanism drivers fail for any reason. e.g. on port creation: https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L1214

Do you mean there should be something similar implemented in the API layer that's generic for all plugins?

[Pav Smirnoff (itwise-net)]

It would make sense as it will protect api from plugins inconsistencies.

On 6 Feb 2018 09:50, "Jakub Libosvar" <email address hidden> wrote:

> We try to do rollbacks in the ML2 code in case mechanism drivers fail
> for any reason. e.g. on port creation:
> https://github.com/openstack/neutron/blob/master/neutron/
> plugins/ml2/plugin.py#L1214
>
> Do you mean there should be something similar implemented in the API
> layer that's generic for all plugins?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746707
>
> Title:
> Neutron DB is made inconsistent when plugin fails
>
> Status in neutron:
> Incomplete
>
> Bug description:
> Numerous issues identified due to the fact that there is no
> transactional consistency between Neutrons own Database operations and
> implementing plugin operations.
>
> Specifically issue identified with the use of NSXV plugin.
>
> Situation#1:
> - Load Balancer being created
> - Healthmonitor created
> - Pool created
> - Provisioning fails due to NSX Manager API error assigning the monitor
> to the pool
> - Yet, details of the pool association are recorded into the Neutron
> Database
> - Subsequent deletion of the pool will fail inside NSXV plugin due to DB
> inconsistency
>
> Situation#2:
> - Similar situation
> - Pool creation may fail due to NSX issues
> - Neutron erroneously records the new pool details despite the
> provisioning errors
> - Deletion attempts of such pool will fail
>
> Situation#3:
> - Security group creation
> - Security group fails to create firewall section in NSXV, but group is
> recorded in Openstack
> - Deletion attempts of this group will fail
>
> General note:
> - Can the MySQL Database transaction wrap be used as a general rule to
> prevent inconsistent DB updates where backing plugin actions result in an
> exception ?
>
> Even though NSXV is clearly identified as a culprit in this case,
> other plugins may easily fail in exactly the same way. End to end API
> call consistency is important, as partial updates may harm operations
> in a variety of ways.
>
> Neutron Version: Mitaka as supplied with VMWARE Integrated Openstack 3.1
> NSX Version: 6.3.0
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/neutron/+bug/1746707/+subscriptions
>

[Isaku Yamahata (yamahata)]

ML2 plugin has precommit callback which is called during neutron db transaction.

ML2 Plugin is okay. It's up to ML2 mech driver how to guarantee consistency between neutron db and its own backend state.
It's another story how well ml2 mech driver is implemented and there are various ways to implement it.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]