Cannot find device "qr-" error message found in logtrace with DVR routers while trying to update arp entry

[Zengfa Gao]

We have running stable/juno with DVR enabled.
During tests, we created router, gateway and instance.

There is one qrouter on one compute node was created with RuntimeError:

Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-086cf9e6-4c43-4b65-b623-fbd5d593f687', 'ip', '-4', 'neigh', 'replace', '10.100.100.13', 'lladdr', 'fa:16:3e:84:fe:e4', 'nud', 'permanent', 'dev', 'qr-00d7d90b-01']
Exit code: 1
Stdout: ''
Stderr: 'Cannot find device "qr-00d7d90b-01"\n'
2015-02-05 20:48:11.834 27031 ERROR neutron.agent.l3_agent [req-2c71f61b-c036-4d90-bcfd-75ffdd5340ff None] DVR: Failed updating arp entry
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent Traceback (most recent call last):
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 1719, in _update_arp_entry
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent device.neigh.add(net.version, ip, mac)
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 515, in add
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent options=[ip_version])
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 247, in _as_root
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent kwargs.get('use_root_namespace', False))
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 79, in _as_root
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent log_fail_as_error=self.log_fail_as_error)
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 91, in _execute
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent log_fail_as_error=log_fail_as_error)
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent File "/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 82, in execute
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent raise RuntimeError(m)
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent RuntimeError:

As the result, all future router update failed as well. When the router was removed, the qrouter namespace was left on the compute node as well because of error:
2015-02-05 20:48:11.834 27031 TRACE neutron.agent.l3_agent Stderr: 'Cannot find device "qr-00d7d90b-01"\n'

Logs also can be read at: http://paste.openstack.org/show/168348/

[Itzik Brown (itzikb1)]

Can you please write the steps to reproduce this case?

[Zengfa Gao (zfgao)]

Here is the script I run several times, I saw it happens once a while:

neutron net-create zengfa-net
netzengfaid=$(neutron net-list | awk '{if($4=="'zengfa-net'"){print $2;}}')
neutron subnet-create zengfa-net 10.100.100.0/24 --name zengfa-subnet
subnetzengfaid=$(neutron subnet-list | awk '{if($4=="'zengfa-subnet'"){print $2;}}')
neutron router-create zengfa-router
routerzengfaid=$(neutron router-list | awk '{if($4=="'zengfa-router'"){print $2;}}')

exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}')
for i in `seq 1 10`; do
    #boot vm, and create floating ip
    nova boot --image cirros --flavor m1.tiny --nic net-id=$netzengfaid cirroszengfa${i}
    cirroszengfaid[i]=$(nova list | awk '{if($4=="'cirroszengfa${i}'"){print $2;}}')
    output=$(neutron floatingip-create $exnetid)
    echo $output
    floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}')
    floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print $4;}}')a
done

# Setup router
neutron router-gateway-set $routerzengfaid $exnetid
neutron router-interface-add zengfa-router $subnetzengfaid
#wait for VM to be running
sleep 30

for i in `seq 1 10`; do
    cirrosfix=$(nova list | awk '{if($4=="'cirroszengfa${i}'"){print $12;}}')
    cirrosfixip=${cirrosfix#*=}
    output=$(neutron port-list | grep ${cirrosfixip})
    echo $output
    portid=$(echo "$output" | awk '{print $2;}')
    neutron floatingip-associate --fixed-ip-address $cirrosfixip ${floatipid[i]} $portid
    neutron floatingip-delete ${floatipid[i]}
    nova delete ${cirroszengfaid[i]}
done

neutron router-interface-delete zengfa-router $subnetzengfaid
neutron router-gateway-clear zengfa-router $netzengfaid
neutron router-delete zengfa-router
neutron subnet-delete $subnetzengfaid
neutron net-delete $netzengfaid

[ZongKai LI (zongkai)]

Hi, Gao ZengFa, could you reproduce this issue again in upstream code?

[Eugene Nikanorov (enikanorov)]

Per previous comment: marking as Incomplete until we know if it's reproducible with upstream or Kilo code

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Swaminathan Vasudevan (swaminathan-vasudevan)]

This bug is still seen in the logtrace.

http://logs.openstack.org/19/225319/2/check/gate-tempest-dsvm-neutron-dvr/42bbfd4/logs/screen-q-l3.txt.gz?level=TRACE#_2015-09-20_23_34_20_361

Confirmed.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Patch posted for review.

https://review.openstack.org/#/c/225514/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230609

[Swaminathan Vasudevan (swaminathan-vasudevan)]

This bug can be closed since it was addressed by another patch.

https://review.openstack.org/#/c/228582/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/225514
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Swaminathan Vasudevan (<email address hidden>) on branch: master
Review: https://review.openstack.org/230609
Reason: Not needed any more. This was addressed by different patch.