puppet-neutron-tripleo-standalone is continuously failing/timing out

Bug #1870257 reported by Takashi Kajinami on 2020-04-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
puppet-neutron
Undecided
Unassigned
tripleo
Critical
Unassigned

Bug Description

puppet-neutron-tripleo-standalone job continuously times out, while it seems to fail in some cases.

We can find that it times out continuously in https://review.opendev.org/#/c/716197/ .

In another patch https://review.opendev.org/#/c/714884/, we can observe the failure in addition to several timeouts.

wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → ussuri-3
tags: added: alert
wes hayutin (weshayutin) wrote :

hrm..

- job:
    name: puppet-neutron-tripleo-standalone
    parent: tripleo-puppet-ci-centos-8-standalone
    description: |
      TripleO standalone Job running Neutron specific Tempest tests.
    vars:
      featureset_override:
        run_tempest: false
        tempest_test_whitelist:
          - 'network'
        use_os_tempest: true

whitelist: 'network'
From a successful job: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8b1/715890/1/check/puppet-neutron-tripleo-standalone/8b12069/logs/undercloud/var/log/tempest/stestr_results.html

So.. I suspect one the network scenarios is getting hung up.

wes hayutin (weshayutin) wrote :

https://review.opendev.org/716823 - just in case .. move to non-voting

create a job in the component pipeline w/ the exact tempest config
https://review.rdoproject.org/r/26213

Takashi Kajinami (kajinamit) wrote :

I believe that the problem is that current tempest scope 'network' is too wide and cause many long running tests.
I don't see any gate jobs testing the same in tripleo, and it doesn't make sense to have wider jobs in puppet side.
We see the patch in puppet-nova to have more specific jobs.
 https://github.com/openstack/puppet-nova/commit/b784174c3cb618b676badb3719ce84a4e19fb0e6

I submitted a change to puppet-neutron to implement the same.
 https://review.opendev.org/#/c/716952

Takashi Kajinami (kajinamit) wrote :

The issue was fixed in the change in puppet-neutron[1], which makes tempest execute less test cases to avoid timeout.
 [1] https://review.opendev.org/#/c/716197/

Changed in puppet-neutron:
status: New → Fix Released
Takashi Kajinami (kajinamit) wrote :

@weshayutin

I think there is nothing we can do from tripleo side, and we can close the bug at tripleo side as WON'T FIX or INVALID.
I'll leave the current status at the moment to ask your confirmation about no more action plans regarding this, but please feel free to close this.

Reviewed: https://review.opendev.org/717040
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=175f063b182c627100404621961371b73b7e6f03
Submitter: Zuul
Branch: stable/train

commit 175f063b182c627100404621961371b73b7e6f03
Author: Takashi Kajinami <email address hidden>
Date: Thu Apr 2 23:53:56 2020 +0900

    Limit scope of puppet-neutron-tripleo-standalone tempest tests

    This is the partial backport of the gate job fix[1] in master, which
    limits scope of tempest tests executed in tripleo job to avoid timeout
    error in gate jobs.

    [1] 6cb1d1af735a0fbd2d811ec40fecbd7d1c987898

    Closes-Bug: #1870257
    Change-Id: I6f72d6be245b5c530de0746fb8b7cec26748bb45

tags: added: in-stable-train

Reviewed: https://review.opendev.org/717041
Committed: https://git.openstack.org/cgit/openstack/puppet-neutron/commit/?id=d6fb669f0d3a91942d9a88b606e0b52bc5ec1773
Submitter: Zuul
Branch: stable/stein

commit d6fb669f0d3a91942d9a88b606e0b52bc5ec1773
Author: Takashi Kajinami <email address hidden>
Date: Thu Apr 2 23:53:56 2020 +0900

    Limit scope of puppet-neutron-tripleo-standalone tempest tests

    This is the partial backport of the gate job fix[1] in master, which
    limits scope of tempest tests executed in tripleo job to avoid timeout
    error in gate jobs.

    [1] 6cb1d1af735a0fbd2d811ec40fecbd7d1c987898

    Conflicts:
     .zuul.yaml

    Closes-Bug: #1870257
    Change-Id: I6f72d6be245b5c530de0746fb8b7cec26748bb45
    (cherry picked from commit 175f063b182c627100404621961371b73b7e6f03)

tags: added: in-stable-stein

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.opendev.org/716823
Reason: we fixed the glitch

This issue was fixed in the openstack/puppet-neutron 16.2.0 release.

wes hayutin (weshayutin) on 2020-04-13
Changed in tripleo:
milestone: ussuri-3 → ussuri-rc3

This issue was fixed in the openstack/puppet-neutron 16.2.1 release.

Brent Eagles (beagles) wrote :

IIUC, this is can be closed on tripleo as well.

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers