Batch update member to pool failed due to new_member is not exist in DB

Bug #1882657 reported by Hemanth Nakkina
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Stein
Fix Released
High
Unassigned
Train
Fix Released
High
Unassigned
Ussuri
Fix Released
High
Unassigned
Victoria
Fix Released
Undecided
Unassigned
octavia (Ubuntu)
Fix Released
Undecided
Hemanth Nakkina
Focal
Fix Released
High
Unassigned
Groovy
Fix Released
Undecided
Hemanth Nakkina

Bug Description

Original Upstream bug: https://storyboard.openstack.org/#!/story/2007581
(The problem is observed in OpenStack Stein)

Code review in progress (Upstream) - https://review.opendev.org/#/c/733670/

==========================================================================

[Impact]
Adding members to loadbalancer in batch mode fails and loadbalancers are
stuck in pending_create state.
This patch retries the read from DB 15 times to check if DB is populated properly or not.

[Test Case]

1. Reproducing the issue

1a. Create 2 or more instances to add as members to loadbalancer
    (The issue can be reproduced if there is load on the machine, on my test environment able to reproduce with 2 members)
1b. Create a loadbalancer, listener, pool and healthmonitor
    openstack loadbalancer create --name lb1 --vip-subnet-id private_subnet
    openstack loadbalancer listener create --name listener1 --protocol HTTP --protocol-port 80 lb1
    openstack loadbalancer pool create --name pool1 --lb-algorithm ROUND_ROBIN --listener listener1 --protocol HTTP
    openstack loadbalancer healthmonitor create --name hm1 --delay 5 --max-retries 4 --timeout 10 --type HTTP --url-path / pool1
1c. Create a token
    AUTH_TOKEN=$(openstack token issue -c id -f value)
1d. Prepare a file members.json based on Request example in the below link
    https://docs.openstack.org/api-ref/load-balancer/v2/?expanded=batch-update-members-detail#batch-update-members
1e. Add members in batch mode
    curl -X PUT -H "Content-Type: application/json" -H "X-Auth-Token: $AUTH_TOKEN" -d @members.json http://<ip>:<port>/v2.0/lbaas/pools/<pool_id>/members
1f. Check loadbalancer show. This will be in pending_create state in non-working case
    openstack loadbalancer show lb1

2. Install the package with fixed code

3. Confirm bug have been fixed

3a. Repeat steps 1a-1f. Loadbalancer should be in ACTIVE state

[Regression Potential]

Upstream CI passed for all the releases including the backported versions.
Installation of new package will restart apache service and hence there will be downtime of octavia service for few milliseconds.

tags: added: sts
Changed in octavia (Ubuntu):
assignee: nobody → Hemanth Nakkina (hemanth-n)
Changed in octavia (Ubuntu Groovy):
status: New → Fix Released
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
description: updated
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Attached debdiffs for focal, uca ussuri, uca train, uca stein.

tags: added: sts-sru-needed
Changed in octavia (Ubuntu Focal):
status: New → Triaged
importance: Undecided → High
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Thank you Hemanth. Uploaded to focal unapproved queue and train/stein staging PPAs.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Hemanth, or anyone else affected,

Accepted octavia into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/octavia/6.0.0-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in octavia (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Hemanth, or anyone else affected,

Accepted octavia into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Hemanth, or anyone else affected,

Accepted octavia into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Hemanth, or anyone else affected,

Accepted octavia into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :

Verified the bug with the fix and it is working as expected. As per the fix, the retry logic have been triggered and corresponding warning messages are printed in octavia-worker log and LB are in Active state. Attached the test case executed for Focal, UCA Ussuri, UCA Train, UCA Stein

Octavia package versions that are verified
Focal 6.0.0-0ubuntu0.20.04.2
UCA Ussuri 6.0.0-0ubuntu0.20.04.2~cloud0
UCA Train 5.0.1-0ubuntu1~cloud1
UCA Stein 4.1.1-0ubuntu1~cloud1

Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
Revision history for this message
Hemanth Nakkina (hemanth-n) wrote :
tags: added: verification-done-focal verification-stein-done verification-train-done verification-ussuri-done
removed: verification-needed-focal verification-stein-needed verification-train-needed verification-ussuri-needed
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package octavia - 6.0.0-0ubuntu0.20.04.2

---------------
octavia (6.0.0-0ubuntu0.20.04.2) focal; urgency=medium

  * Fix batch creation of new members failure due to timeouts (LP: #1882657)
    - d/p/fix-batch-member-create-for-v1-amphora-driver.patch

 -- Hemanth Nakkina <email address hidden> Wed, 08 Jul 2020 09:43:39 +0530

Changed in octavia (Ubuntu Focal):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for octavia has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for octavia has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package octavia - 6.0.0-0ubuntu0.20.04.2~cloud0
---------------

 octavia (6.0.0-0ubuntu0.20.04.2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 octavia (6.0.0-0ubuntu0.20.04.2) focal; urgency=medium
 .
   * Fix batch creation of new members failure due to timeouts (LP: #1882657)
     - d/p/fix-batch-member-create-for-v1-amphora-driver.patch

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for octavia has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package octavia - 5.0.1-0ubuntu1~cloud1
---------------

 octavia (5.0.1-0ubuntu1~cloud1) bionic-train; urgency=medium
 .
   * Fix batch creation of new members failure due to timeouts (LP: #1882657)
     - d/p/fix-batch-member-create-for-v1-amphora-driver.patch

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for octavia has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package octavia - 4.1.1-0ubuntu1~cloud1
---------------

 octavia (4.1.1-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   * Fix batch creation of new members failure due to timeouts (LP: #1882657)
     - d/p/fix-batch-member-create-for-v1-amphora-driver.patch

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.