ovs-vswitchd core dump in ovsfw jobs

[YAMAMOTO Takashi]

eg. http://logs.openstack.org/76/503076/12/check/neutron-tempest-ovsfw/c350c10/logs/openvswitch/ovs-vswitchd.txt.gz

2018-01-11T09:12:37.323Z|01923|bridge|INFO|bridge br-int: deleted interface tap251a9cf5-3b on port 154
2018-01-11T09:12:37.327Z|01924|bridge|WARN|could not open network device tap251a9cf5-3b (No such device)
2018-01-11T09:12:37.476Z|01925|bridge|WARN|could not open network device tap251a9cf5-3b (No such device)
2018-01-11T09:12:37.479Z|01926|bridge|INFO|bridge br-int: deleted interface tapb73da219-3c on port 165
2018-01-11T09:12:37.484Z|01927|bridge|WARN|could not open network device tapb73da219-3c (No such device)
2018-01-11T09:12:37.487Z|01928|bridge|WARN|could not open network device tap251a9cf5-3b (No such device)
2018-01-11T09:12:37.580Z|01929|bridge|WARN|could not open network device tapb73da219-3c (No such device)
2018-01-11T09:12:37.875Z|01930|bridge|INFO|bridge br-int: added interface tapb73da219-3c on port 166
2018-01-11T09:12:38.267Z|01931|bridge|INFO|bridge br-int: added interface qr-e5ef3d2d-42 on port 167
2018-01-11T09:12:38.332Z|01932|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-e5ef3d2d-42 device failed: No such device
2018-01-11T09:12:39.150Z|01933|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2018-01-11T09:12:39.166Z|01934|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2018-01-11T09:12:39.202Z|01935|connmgr|INFO|br-int<->unix: 2 flow_mods in the last 0 s (2 deletes)
2018-01-11T09:12:39.219Z|01936|connmgr|INFO|br-int<->unix: 56 flow_mods in the last 0 s (56 adds)
2018-01-11T09:12:39.636Z|01937|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2018-01-11T09:12:39.643Z|01938|connmgr|INFO|br-int<->unix: 1 flow_mods in the last 0 s (1 deletes)
2018-01-11T09:12:39.652Z|01939|connmgr|INFO|br-int<->unix: 2 flow_mods in the last 0 s (2 deletes)
2018-01-11T09:12:40.866Z|01940|connmgr|INFO|br-int<->unix: 2 flow_mods in the last 0 s (2 deletes)
2018-01-11T09:12:40.875Z|01941|bridge|INFO|bridge br-int: deleted interface qr-e5ef3d2d-42 on port 167
2018-01-11T09:12:40.909Z|01942|util|EMER|../lib/netdev.c:2281: assertion ifidx failed in netdev_ports_remove()
2018-01-11T09:12:41.251Z|00002|daemon_unix(monitor)|ERR|1 crashes: pid 7122 died, killed (Aborted), core dumped, restarting

[Jakub Libosvar (libosvar)]

shouldn't this be reported against openvswitch?

[IWAMOTO Toshihiro (iwamoto)]

neutron-tempest-ovsfw has 70-100% failure rate probably due to this.

[IWAMOTO Toshihiro (iwamoto)]

From dpkg.txt.gz:
    openvswitch-switch 2.8.0-0ubuntu2~cloud0

Might have been already fixed in ovs git repo.

commit 3a23430b573e8abfbae46493059fa7366b36f584
Author: Ashish Varma <email address hidden>
Date: Mon Nov 6 12:17:45 2017 -0800

    netdev, dpif: fix the crash/assert on port delete

    a crash is seen in "netdev_ports_remove" when an interface is deleted and ad
ded
    back in the system and when the interface is part of a bridge configuration.
    e.g. steps:
      create a tap0 interface using "ip tuntap add.."
      add the tap0 interface to br0 using "ovs-vsctl add-port.."
      delete the tap0 interface from system using "ip tuntap del.."
      add the tap0 interface back in system using "ip tuntap add.."
                           (this changes the ifindex of the interface)
      delete tap0 from br0 using "ovs-vsctl del-port.."

    In the function "netdev_ports_insert", two hmap entries were created for
    mapping "portnum -> netdev" and "ifindex -> portnum".
    When the interface is deleted from the system, the "netdev_ports_remove"
    function is not getting called and the old ifindex entry is not getting
    cleaned up from the "ifindex_to_port" hmap.

    As part of the fix, added function "dpif_port_remove" which will call
    "netdev_ports_remove" in the path where the interface deletion from the syst
em
    is detected.
    Also, in "netdev_ports_remove", added the code where the "ifindex_to_port_da
ta"
    (ifindex -> portnum map node) is getting freed when the ifindex is not
    available any more. (as the interface is already deleted.)

    VMware-BZ: #1975788
    Signed-off-by: Ashish Varma <email address hidden>
    Signed-off-by: Ben Pfaff <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/542596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/542596
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=46d693530b97581998313fb307201a3afd7da112
Submitter: Zuul
Branch: master

commit 46d693530b97581998313fb307201a3afd7da112
Author: IWAMOTO Toshihiro <email address hidden>
Date: Fri Feb 9 14:21:02 2018 +0900

    Compile ovs for ovsfw to avoid ovs-vswitchd core dumps

    Change-Id: I750224f7495aa46634bec1211599953cbbd4d1df
    Closes-bug: #1742863

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/543963

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543963
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=535b6d2b41c69a0e8131d2935a93cca2e577299b
Submitter: Zuul
Branch: stable/queens

commit 535b6d2b41c69a0e8131d2935a93cca2e577299b
Author: IWAMOTO Toshihiro <email address hidden>
Date: Fri Feb 9 14:21:02 2018 +0900

    Compile ovs for ovsfw to avoid ovs-vswitchd core dumps

    Change-Id: I750224f7495aa46634bec1211599953cbbd4d1df
    Closes-bug: #1742863
    (cherry picked from commit 46d693530b97581998313fb307201a3afd7da112)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.