2.8.0: test failure under s390x

[James Page]

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 4 tests were run,
1 failed unexpectedly.

## ------------------------ ##
## Summary of the failures. ##
## ------------------------ ##
Failed tests:
openvswitch 2.8.0 test suite test groups:

 NUM: FILE-NAME:LINE TEST-GROUP-NAME
      KEYWORDS

 2222: stp.at:479 STP - flush the fdb and mdb when topology changed

[James Page (james-page)]

Digging into this further, it would appear that the STP convergence is taking longer on s390x; if we time warp a further 30 seconds before checking the fdb table for br0, the entry for the removed p2 device has gone.

[James Page (james-page)]

Reading logs between a successful run on x86 and a failed run on s390x, it would appear that stp never sees the topology change on br0 as p2 is removed - this log message is missing from the s390x logs:

   br0: detected topology change.

so I suspect the fdb cache ends up hitting some sort of age timeout before it gets flushed, rather than being activity flushed as a result of the topology change.

[James Page (james-page)]

That might be a red herring; however I do see:

2017-10-18T10:15:49.857Z|05086|bridge|INFO|bridge br0: deleted interface p2 on port 2
2017-10-18T10:15:49.857Z|05087|ofproto_dpif|DBG|port p2: STP state changed from forwarding to disabled

vs (on x86)

2017-10-18T10:25:04.902Z|04382|bridge|INFO|bridge br0: deleted interface p2 on port 2
2017-10-18T10:25:04.902Z|04383|ofproto_dpif|DBG|port p2: STP state changed from forwarding to disabled
2017-10-18T10:25:04.903Z|04384|ofproto_dpif|DBG|port p1: STP state changed from forwarding to listening

not the missing state transition for port p1 on s390x

[James Page (james-page)]

I was able to reproduce this issue on 2.8.0 and 2.7.1 but not in 2.5.2; some bisection later pointed to:

  https://github.com/openvswitch/ovs/commit/013ebfe62d4625afb5bc669e3a32b5a1a236a9be

however I think that just uncovers the actual issue, rather than being the root cause as that applies on all archs.

[James Page (james-page)]

Bumping priority - this is blocking other SRU activity which needs a clear runway; s390x ftbfs in artful and xenial today, so we need to get this resolved.

[Adam Conrad (adconrad)]

Hello James, or anyone else affected,

Accepted openvswitch into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.8.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package openvswitch - 2.8.0-0ubuntu2

---------------
openvswitch (2.8.0-0ubuntu2) artful; urgency=medium

  [ James Page ]
  * d/p/s390x-stp-timeout.patch: Increase STP sync wait time for
    'STP - flush the fdb and mdb when topology changed' test as this
    reliable takes longer than 36 seconds on s390x (LP: #1722799).

  [ Frode Nordahl ]
  * d/openvswitch-switch.postinst: Do not modify
    /etc/default/openvswitch-switch as this file is now managed
    as a configuration file by dpkg (LP: #1723480).

 -- James Page <email address hidden> Thu, 19 Oct 2017 11:04:37 +0100

[Corey Bryant (corey.bryant)]

Hello James, or anyone else affected,

Accepted openvswitch into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Regression testing with OpenStack tempest has completed successfully for xenial-pike-proposed.

smoke tests with next charms:

======
Totals
======
Ran: 102 tests in 1361.9098 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 616.0666 sec.

smoke tests with stable charms:

======
Totals
======
Ran: 102 tests in 1361.9098 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 616.0666 sec.

tags: added: verification-pike-done
removed: verification-pike-needed

[James Page (james-page)]

The verification of the Stable Release Update for openvswitch has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package openvswitch - 2.8.0-0ubuntu2~cloud0
---------------

 openvswitch (2.8.0-0ubuntu2~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 openvswitch (2.8.0-0ubuntu2) artful; urgency=medium
 .
   [ James Page ]
   * d/p/s390x-stp-timeout.patch: Increase STP sync wait time for
     'STP - flush the fdb and mdb when topology changed' test as this
     reliable takes longer than 36 seconds on s390x (LP: #1722799).
 .
   [ Frode Nordahl ]
   * d/openvswitch-switch.postinst: Do not modify
     /etc/default/openvswitch-switch as this file is now managed
     as a configuration file by dpkg (LP: #1723480).