grenade-dvr-multinode job fails

Bug #1791989 reported by Slawek Kaplonski on 2018-09-11
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Slawek Kaplonski
Brian Haley (brian-haley) wrote :

Suspecting this might be due to the eventlet bump, I created a revert and a dependent neutron patch to test the theory:

https://review.openstack.org/#/c/601660/
https://review.openstack.org/#/c/601652/

The neutron change passed zuul on the first try and is in recheck. If it passes recheck a few more times I'll propose a proper revert. Right now it's hard to know if it's the correct fix.

Brian Haley (brian-haley) wrote :

Re-check of the above patch shows a neutron-grenade-dvr-multinode test failure, so probably not due to eventlet bump.

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/602155
Reason: this was wrong

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/602156

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/602178

Change abandoned by Brian Haley (<email address hidden>) on branch: master
Review: https://review.openstack.org/602688

Reviewed: https://review.openstack.org/602097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=db7e19d2ba3fcde0510be0094a4851178fae578b
Submitter: Zuul
Branch: master

commit db7e19d2ba3fcde0510be0094a4851178fae578b
Author: Brian Haley <email address hidden>
Date: Wed Sep 12 10:54:47 2018 -0600

    Fix pylint warning with eventlet > 0.22.0

    When eventlet was bumped up to 0.24.1 in upper-constraints [1]
    it started generating a pylint warning, since neutron has
    support for older versions such as 0.22.0 that take a different
    number of arguments. Just silence the warning.

    Also, in order to get this through the gate, the
    neutron-grenade-dvr-multinode check job is being temporarily
    set to non-voting and removed from the gate. That change
    should be reverted as soon as we fix bug 1791989.

    [1] https://review.openstack.org/#/c/589382/

    Change-Id: I169ecb6f33b670c5469919214a32645cd9eda206
    Closes-Bug: #1791178
    Related-Bug: #1791989

Changed in neutron:
assignee: nobody → Slawek Kaplonski (slaweq)

Related fix proposed to branch: master
Review: https://review.openstack.org/603484

Slawek Kaplonski (slaweq) wrote :

I suspected that it may be some issue with packages like libvirt, qemu or openvswitch but as same issue happens also on patches for stable/pike branch (see http://logs.openstack.org/48/603048/1/check/neutron-grenade-dvr-multinode/8686b27/logs/) and all those packages are in different versions then on master I think that it's something else.

tags: added: neutron-proactive-backport-potential
Slawek Kaplonski (slaweq) wrote :

I used query like: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22'%5BFail%5D%20Couldn'%5C%5C''t%20ping%20server'%5C%22 to find such issues in the gates.
What I can say is that this issue happens only on:
- master branch - around 85% of cases
- stable/pike - around 15% of cases

There is no any issue like that on stable/ocata, stable/queens or stable/rocky branch. I was suspecting packages like libvirt, qemu or openvswitch but all of them are in completely different versions on stable/pike jobs and are failing also so it's not that IMO.

I'm really puzzled and I have no idea what could cause such issue.

Related fix proposed to branch: master
Review: https://review.openstack.org/606208

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/603484

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/603483

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/602156
Reason: issue is finally resolved

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/606208
Reason: Issue already fixed :)

Reviewed: https://review.openstack.org/609437
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f44e5bcfb2a4e38b57d61f56d0cc980530bda9bc
Submitter: Zuul
Branch: master

commit f44e5bcfb2a4e38b57d61f56d0cc980530bda9bc
Author: Slawek Kaplonski <email address hidden>
Date: Wed Oct 10 16:50:55 2018 +0200

    Make grenade-multinode-dvr job voting again

    This job was set to be non-voting and was removed from
    gate queue in [1]
    It looks that patch [2] which was merged to stable/rocky
    recently fixed this issue and this job is working fine again,
    so lets make it voting and gating again.

    [1] https://review.openstack.org/#/c/602097/
    [2] https://review.openstack.org/#/c/595490/

    Change-Id: Ied3b7c439056134388b16742448cd1da7ece04cf
    Related-Bug: #1791989

Slawek Kaplonski (slaweq) wrote :

Problem solved by https://review.openstack.org/#/c/595490/ in stable/rocky. Job is again voting and gating so this bug can be closed now.

Changed in neutron:
status: Confirmed → Fix Committed
status: Fix Committed → Fix Released
tags: removed: neutron-proactive-backport-potential

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/608259

Reviewed: https://review.openstack.org/607685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ac5815a110f0033c1a217481822fc0c72c1ba16c
Submitter: Zuul
Branch: master

commit ac5815a110f0033c1a217481822fc0c72c1ba16c
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

    Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989

tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential

Reviewed: https://review.openstack.org/613536
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e92ea0bade4a381988b843973ec3e477e76e2fe
Submitter: Zuul
Branch: stable/ocata

commit 1e92ea0bade4a381988b843973ec3e477e76e2fe
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

    Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags: added: in-stable-ocata

Reviewed: https://review.openstack.org/613534
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=861dfbd0d2ef8488928144cbd9066172501fbe85
Submitter: Zuul
Branch: stable/queens

commit 861dfbd0d2ef8488928144cbd9066172501fbe85
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

    Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags: added: in-stable-queens

Reviewed: https://review.openstack.org/608259
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fc944d3e8ec913aea983ffbf61d4e88e960e9ad4
Submitter: Zuul
Branch: stable/rocky

commit fc944d3e8ec913aea983ffbf61d4e88e960e9ad4
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

    Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags: added: in-stable-rocky

Reviewed: https://review.openstack.org/613535
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=16c2d64bdc97ecadd857c74c753369085d98f625
Submitter: Zuul
Branch: stable/pike

commit 16c2d64bdc97ecadd857c74c753369085d98f625
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

    Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags: added: in-stable-pike
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers