neutron

grenade-dvr-multinode job fails

Bug #1791989 reported by Slawek Kaplonski on 2018-09-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Slawek Kaplonski

Bug Description

This jobs is failing quite often in last days.
In all cases which I checked there was similar issue in grenade log, for example: http://logs.openstack.org/65/600565/2/gate/neutron-grenade-dvr-multinode/07ef603/logs/grenade.sh.txt.gz#_2018-09-11_16_12_33_959

It hits more than 70 times during last 7 days:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22neutron-grenade-dvr-multinode%5C%22%20AND%20build_status%3AFAILURE%20AND%20message%3A%5C%22die%2067%20'%5BFail%5D%20Couldn'%5C%5C''t%20ping%20server%5C%22

Tags:

Revision history for this message

Brian Haley (brian-haley) wrote on 2018-09-11:

Suspecting this might be due to the eventlet bump, I created a revert and a dependent neutron patch to test the theory:

https://review.openstack.org/#/c/601660/
https://review.openstack.org/#/c/601652/

The neutron change passed zuul on the first try and is in recheck. If it passes recheck a few more times I'll propose a proper revert. Right now it's hard to know if it's the correct fix.

Revision history for this message

Brian Haley (brian-haley) wrote on 2018-09-11:

Re-check of the above patch shows a neutron-grenade-dvr-multinode test failure, so probably not due to eventlet bump.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/602155

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Change abandoned on neutron (stable/rocky)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/602155
Reason: this was wrong

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/602156

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/602156

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/602178

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-12: Change abandoned on neutron (stable/rocky)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/602178

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-14: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/602688

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-14: Change abandoned on neutron (master)

#10

Change abandoned by Brian Haley (<email address hidden>) on branch: master
Review: https://review.openstack.org/602688

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-15: Related fix merged to neutron (master)

#11

Reviewed: https://review.openstack.org/602097
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=db7e19d2ba3fcde0510be0094a4851178fae578b
Submitter: Zuul
Branch: master

commit db7e19d2ba3fcde0510be0094a4851178fae578b
Author: Brian Haley <email address hidden>
Date: Wed Sep 12 10:54:47 2018 -0600

Fix pylint warning with eventlet > 0.22.0

    When eventlet was bumped up to 0.24.1 in upper-constraints [1]
    it started generating a pylint warning, since neutron has
    support for older versions such as 0.22.0 that take a different
    number of arguments. Just silence the warning.

    Also, in order to get this through the gate, the
    neutron-grenade-dvr-multinode check job is being temporarily
    set to non-voting and removed from the gate. That change
    should be reverted as soon as we fix bug 1791989.

[1] https://review.openstack.org/#/c/589382/

    Change-Id: I169ecb6f33b670c5469919214a32645cd9eda206
    Closes-Bug: #1791178
    Related-Bug: #1791989

Slawek Kaplonski (slaweq) on 2018-09-18

Changed in neutron:
assignee:	nobody → Slawek Kaplonski (slaweq)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-18: Related fix proposed to neutron (master)

#12

Related fix proposed to branch: master
Review: https://review.openstack.org/603483

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-18:

#13

Related fix proposed to branch: master
Review: https://review.openstack.org/603484

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-09-21:

#14

I suspected that it may be some issue with packages like libvirt, qemu or openvswitch but as same issue happens also on patches for stable/pike branch (see http://logs.openstack.org/48/603048/1/check/neutron-grenade-dvr-multinode/8686b27/logs/) and all those packages are in different versions then on master I think that it's something else.

Bernard Cafarelli (bcafarel) on 2018-09-21

tags:

added: neutron-proactive-backport-potential

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-09-21:

#15

I used query like: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22'%5BFail%5D%20Couldn'%5C%5C''t%20ping%20server'%5C%22 to find such issues in the gates.
What I can say is that this issue happens only on:
- master branch - around 85% of cases
- stable/pike - around 15% of cases

There is no any issue like that on stable/ocata, stable/queens or stable/rocky branch. I was suspecting packages like libvirt, qemu or openvswitch but all of them are in completely different versions on stable/pike jobs and are failing also so it's not that IMO.

I'm really puzzled and I have no idea what could cause such issue.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-28:

#16

Related fix proposed to branch: master
Review: https://review.openstack.org/606208

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-28: Change abandoned on neutron (master)

#17

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/603484

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-09-28:

#18

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/603483

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-03: Related fix proposed to neutron (master)

#19

Related fix proposed to branch: master
Review: https://review.openstack.org/607685

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-05: Related fix proposed to neutron (stable/rocky)

#20

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/608259

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-10: Related fix proposed to neutron (master)

#21

Related fix proposed to branch: master
Review: https://review.openstack.org/609437

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-11: Change abandoned on neutron (master)

#22

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/602156
Reason: issue is finally resolved

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-11:

#23

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/606208
Reason: Issue already fixed :)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-12: Related fix merged to neutron (master)

#24

Reviewed: https://review.openstack.org/609437
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f44e5bcfb2a4e38b57d61f56d0cc980530bda9bc
Submitter: Zuul
Branch: master

commit f44e5bcfb2a4e38b57d61f56d0cc980530bda9bc
Author: Slawek Kaplonski <email address hidden>
Date: Wed Oct 10 16:50:55 2018 +0200

Make grenade-multinode-dvr job voting again

    This job was set to be non-voting and was removed from
    gate queue in [1]
    It looks that patch [2] which was merged to stable/rocky
    recently fixed this issue and this job is working fine again,
    so lets make it voting and gating again.

[1] https://review.openstack.org/#/c/602097/
[2] https://review.openstack.org/#/c/595490/

Change-Id: Ied3b7c439056134388b16742448cd1da7ece04cf
Related-Bug: #1791989

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2018-10-12:

#25

Problem solved by https://review.openstack.org/#/c/595490/ in stable/rocky. Job is again voting and gating so this bug can be closed now.

Changed in neutron:
status:	Confirmed → Fix Committed
status:	Fix Committed → Fix Released

Slawek Kaplonski (slaweq) on 2018-10-12

tags:

removed: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-12: Change abandoned on neutron (stable/rocky)

#26

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.openstack.org/608259

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-16: Related fix merged to neutron (master)

#27

Reviewed: https://review.openstack.org/607685
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ac5815a110f0033c1a217481822fc0c72c1ba16c
Submitter: Zuul
Branch: master

commit ac5815a110f0033c1a217481822fc0c72c1ba16c
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
Related-Bug: #1791989

Bernard Cafarelli (bcafarel) on 2018-10-19

tags:	added: neutron-proactive-backport-potential
tags:	added: neutron-easy-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-26: Related fix proposed to neutron (stable/queens)

#28

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/613534

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-26: Related fix proposed to neutron (stable/pike)

#29

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/613535

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-26: Related fix proposed to neutron (stable/ocata)

#30

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/613536

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-29: Related fix merged to neutron (stable/ocata)

#31

Reviewed: https://review.openstack.org/613536
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e92ea0bade4a381988b843973ec3e477e76e2fe
Submitter: Zuul
Branch: stable/ocata

commit 1e92ea0bade4a381988b843973ec3e477e76e2fe
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-29: Related fix merged to neutron (stable/queens)

#32

Reviewed: https://review.openstack.org/613534
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=861dfbd0d2ef8488928144cbd9066172501fbe85
Submitter: Zuul
Branch: stable/queens

commit 861dfbd0d2ef8488928144cbd9066172501fbe85
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-10-31: Related fix merged to neutron (stable/rocky)

#33

Reviewed: https://review.openstack.org/608259
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fc944d3e8ec913aea983ffbf61d4e88e960e9ad4
Submitter: Zuul
Branch: stable/rocky

commit fc944d3e8ec913aea983ffbf61d4e88e960e9ad4
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-02: Related fix merged to neutron (stable/pike)

#34

Reviewed: https://review.openstack.org/613535
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=16c2d64bdc97ecadd857c74c753369085d98f625
Submitter: Zuul
Branch: stable/pike

commit 16c2d64bdc97ecadd857c74c753369085d98f625
Author: Brian Haley <email address hidden>
Date: Wed Oct 3 14:35:05 2018 -0400

Add permanent ARP entries for DVR fip/qrouter veth pair

    Since we know the IP and MAC addresses of both sides of the
    fip/qrouter namespace veth pair device, just add permanent
    ARP entries for them.

    Change-Id: I6193b00681dfb79222eedfd00c89620321ac1b4f
    Related-Bug: #1791989
    (cherry picked from commit ac5815a110f0033c1a217481822fc0c72c1ba16c)

tags:

added: in-stable-pike

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.