gratuitous arping causes exception filling up logs with errors on Ubuntu14.04

[Arun]

Connecting an external network to a router causes l3-agent to send out gratuitous arp packets to to the external network gateway IP in order to pre-populate the mac table without expecting a response. Similar behavior when associating floating IPs. Arping utility on CentOS7.2 returns 0 when no response but returns 1 on Ubuntu14.04 which causes an exception and thus a Traceback in the log files.

Changing arping command call to not check return code status fixes this issue and that is the proposed fix.

Pre-conditions: Router with external network attached.

Step-by-step reproduction steps:
1) Create a router
2) Attach an external network as gateway.
3) Attach a tenant network to the router.
4) Associate a Floating IP to a VM instance powered on in that tenant network.

Expected output: No errors in logs. Seen on CentOS7.2
Actual output: Traceback in l3 log. Seen on Ubuntu14.04
http://paste.openstack.org/show/496284/

Version: Openstack Liberty (Tag: 7.0.2). Ubuntu14.04.
uname -a
Linux ubuntu01 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Services running: l3-agent in dvr mode, ovs-agent, dhcp-agent, nova-compute.

Perceived Severity: Medium (Causes issues with active monitoring)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/313177

[Ryan Moats (rmoats)]

Brian/Oleg - can you see if this is really a problem with master? Logstash doesn't show me anything that looks like the reported error signature in the last seven days...

[Hong Hui Xiao (xiaohhui)]

Actually, I just tried in ubuntu, the arp return 0, is there something I missed out?

ubuntu@ubuntu:~$ arping -A -I eth0 9.5.126.181 -w 1
ARPING 9.5.126.181 from 9.5.126.181 eth0
Sent 2 probes (2 broadcast(s))
Received 0 response(s)

ubuntu@ubuntu:~$ echo $?
0

ubuntu@ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.1 LTS
Release: 14.04
Codename: trusty

[Arun (sarun87)]

Sorry for the delay. This is my first bug/review with Openstack. Interesting , I see different outputs at my end. Get a return code of 0 only when arping receives packets.

root@ubuntu01:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
root@ubuntu01:~# arping -A -I eth0 10.4.0.1 -c 2 -w 1
ARPING 10.4.0.1

--- 10.4.0.1 statistics ---
2 packets transmitted, 0 packets received, 100% unanswered (0 extra)
root@ubuntu01:~# echo $?
1
root@ubuntu01:~# arping -A -I eth0 10.4.0.1 -c 2
ARPING 10.4.0.1

--- 10.4.0.1 statistics ---
2 packets transmitted, 0 packets received, 100% unanswered (0 extra)
root@ubuntu01:~# echo $?
1
root@ubuntu01:~# arping -I eth0 10.4.0.1 -c 2
ARPING 10.4.0.1
60 bytes from 00:18:0a:86:da:04 (10.4.0.1): index=0 time=3.338 msec
60 bytes from 00:18:0a:86:da:04 (10.4.0.1): index=1 time=2.463 msec

--- 10.4.0.1 statistics ---
2 packets transmitted, 2 packets received, 0% unanswered (0 extra)
root@ubuntu01:~# echo $?
0
root@ubuntu01:~# arping
ARPing 2.11, by Thomas Habets <email address hidden>
usage: arping [ -0aAbdDeFpqrRuv ] [ -w <us> ] [ -S <host/ip> ]
              [ -T <host/ip ] [ -s <MAC> ] [ -t <MAC> ] [ -c <count> ]
              [ -i <interface> ] <host/ip/MAC | -B>
For complete usage info, use --help or check the manpage.
root@ubuntu01:~#

@Hong Would it be a problem with my arping Binary? Could you share your arping version?

[Sean M. Collins (scollins)]

Coming across this in a 3rd party CI.

http://eavesdrop.openstack.org/irclogs/%23openstack-dev/%23openstack-dev.2016-05-12.log.html#t2016-05-12T19:08:17

[Sreekumar S (sreesiv)]

This is updated with a new PS in https://review.openstack.org/313177 to add extra OK code.

[Arun (sarun87)]

Sreekumar, I had parked this bug without an update to verify if mine was a one off Hong Hui Xiao (xiaohhui) said he could not repro this. Have you been able to reproduce this behavior on your setup?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313177
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc9ce5189db4f3300c38b36dd28a98d64ca7a1fe
Submitter: Jenkins
Branch: master

commit cc9ce5189db4f3300c38b36dd28a98d64ca7a1fe
Author: Arun Sriraman <email address hidden>
Date: Thu May 5 22:18:24 2016 +0000

    Changing arping command execute to accept 1 as extra OK code

    Connecting any network to a router causes l3-agent to send out
    gratuitous ARP packets that do not require a response. On Ubuntu 14.04
    sometimes this command returns 1, which causes it to falsely report
    arping command failure. This is now added to the extra OK codes.

    Change-Id: I6299e936d4e4e7fd7497d23224e7abf605a99c15
    Closes-bug: #1578842

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/397043

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/397049

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/397052

[Sreekumar S (sreesiv)]

This doesn't seem to meet the criteria for Liberty as it's at the 12-month old timeframe:
http://docs.openstack.org/project-team-guide/stable-branches.html#support-phases

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sreekumar S (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/397052
Reason: As per Brian Haley,

This doesn't seem to meet the criteria for Liberty as it's at the 12-month old timeframe:
http://docs.openstack.org/project-team-guide/stable-branches.html#support-phases

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/397043
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=869234641dfc06a346a81f736f9d4bc12e09293c
Submitter: Jenkins
Branch: stable/newton

commit 869234641dfc06a346a81f736f9d4bc12e09293c
Author: Arun Sriraman <email address hidden>
Date: Thu May 5 22:18:24 2016 +0000

    Changing arping command execute to accept 1 as extra OK code

    Connecting any network to a router causes l3-agent to send out
    gratuitous ARP packets that do not require a response. On Ubuntu 14.04
    sometimes this command returns 1, which causes it to falsely report
    arping command failure. This is now added to the extra OK codes.

    Conflicts:
     neutron/agent/linux/ip_lib.py

    Change-Id: I6299e936d4e4e7fd7497d23224e7abf605a99c15
    Closes-bug: #1578842
    (cherry picked from commit cc9ce5189db4f3300c38b36dd28a98d64ca7a1fe)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/397049
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=339e1747383468f121244c7db628a9523ed0d647
Submitter: Jenkins
Branch: stable/mitaka

commit 339e1747383468f121244c7db628a9523ed0d647
Author: Arun Sriraman <email address hidden>
Date: Thu May 5 22:18:24 2016 +0000

    Changing arping command execute to accept 1 as extra OK code

    Connecting any network to a router causes l3-agent to send out
    gratuitous ARP packets that do not require a response. On Ubuntu 14.04
    sometimes this command returns 1, which causes it to falsely report
    arping command failure. This is now added to the extra OK codes.

    Conflicts:
            neutron/agent/linux/ip_lib.py

    Change-Id: I6299e936d4e4e7fd7497d23224e7abf605a99c15
    Closes-bug: #1578842
    (cherry picked from commit cc9ce5189db4f3300c38b36dd28a98d64ca7a1fe)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 8.4.0 release.