nova interface-attach command removes pre-existing neutron ports from the environment if it fails to attach to an instance _even_ where '--port-id' has been specified

[Michael Cunningham]

Problem description:
The nova interface-attach command removes pre-existing neutron ports from the environment if it fails to attach to an instance _even_ where '--port-id' has been specified. This behaviour was introduced by fixing bug #1338551 [1].

Steps to reproduce:
1) create a new neutron port
  $ neutron port-create --name <port-name> <network-name>
2) boot an instance (make sure to specify a keypair and check sec groups for ssh connectivity to the instance)
  $ nova boot ...
3) [OPTIONAL] add/remove the port several times over to prove the funtionality is working OK.
  $ nova interface-attach --port-id <port-id> <instance-name>
  $ nova interface-detach <instance-name> <port-id>
4) simulate a kernel crash on the instance, as this should cause a scenario where an interface attach will fail (ssh connectivity is assumed for this step)
  $ ssh <instance-ip> "sudo kill -11 1" # OR execute 'echo c > /proc/sysrq-trigger' while connected to the instance
  4a. Verify the kernel has actually crashed
      $ nova console-log <instance-name>
5) try to attach the port while the instance is still crashed # **note** if the port hasn't been attached before (i.e. you skipped step 3, it may succeed initially then it will fail on subsequent attach attempts). Also, at this point it should not matter if the port is still attached to the instance.
  $ nova interface-attach --port-id <port-id> <instance-name>

Errors observed:
$ nova interface-attach --port-id <port-id> <instance-name>
ERROR: Failed to attach interface (HTTP 500) (Request-ID: req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

Expected results:
The port should still exist after failure in this scenario.

Actual results:
'neutron port-list' will no longer show the port. It has been removed.
The port is removed from the environment and therefore is no longer available.

Snippet from /var/log/nova/nova-compute.log

[instance: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx] attaching network adapter failed.
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1263, in attach_interface
     virt_dom.attachDeviceFlags(cfg.to_xml(), flags)
   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
     result = proxy_call(self._autowrap, f, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
     rv = execute(f, *args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
     six.reraise(c, e, tb)
   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
     rv = meth(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in attachDeviceFlags
     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
 libvirtError: Unable to create tap device tapxxxxxxxx-xx: Device or resource busy
 attach interface failed , try to deallocate port xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, reason: Failed to attach network adapter device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 Exception during message handling: Failed to attach network adapter device to xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Full error:
http://pastebin.ubuntu.com/15471511/

$ sudo apt-cache policy nova-compute
nova-compute:
  Installed: 1:2015.1.2-0ubuntu2~cloud0

Ubuntu 14.04.4 LTS

Why does this matter:
As specified in [1], where a port has been attached using --net-id option it is automatically created before attaching to the VM. Therefore, it is the correct behaviour to cleanup after a failure to attach.
Where "--port-id" has been specified, it should not be assumed that it was auto created, it has been specifically created and therefore may have pre-existed the VM, this means the port should be re-usable if desired and therefore should not be cleaned up in the case of attach failure. When the port has been pre-created and '--port-id' is specified in the interface-attach command, if the action fails to attach it should be handled without being removed from the environment and exist for re-assignment to another instance or for retry to the original instance once it has recovered from it's failure.

This behaviour is confirmed on both Kilo and Liberty.

Related bugs:
[1] https://bugs.launchpad.net/nova/+bug/1338551

[Sreenivas (sreenivas-pothukanoori)]

I tried to reproduce this issue on Kilo & Liberty versions, but I haven't observed the reported issue.

I have followed the below steps:
1. Logged in as admin user using "source admin-openrc.sh".

2. Created network port using "neutron port-create --name port-id net-name"
+--------------------------------------+-----------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+-----------+-------------------+--------------------------------------------------------------------------------------+
| be34ff0c-ca67-4d41-9fd8-12eb9caa46c1 | demoPort2 | fa:16:3e:29:c0:09 | {"subnet_id": "caed0666-05f3-414a-b5b7-4eceeb0ca5ae", "ip_address": "172.16.1.36"} |
+--------------------------------------+-----------+-------------------+--------------------------------------------------------------------------------------+

3. Created security group with the following details:
+-------------+-----------+---------+-----------+--------------+
| IP Protocol | From Port | To Port | IP Range | Source Group |
+-------------+-----------+---------+-----------+--------------+
| icmp | -1 | -1 | 0.0.0.0/0 | |
| | | | | default |
| | | | | default |
| tcp | 22 | 22 | 0.0.0.0/0 | |
+-------------+-----------+---------+-----------+--------------+

4.Launched the VM using "nova boot --flavor m1.tiny --image cirros --nic net-id=network-id --security-group default --key-name mykey DemoVM".

5.Attached interface to the DemoVM and detached it. Performed this step twice using the below commands:
nova interface-attach --port-id <port-id> <instance-name>
nova interface-detach <instance-name> <port-id>

6. connected to the VM instance console with VM IP using "ip netns exec qdhcp-net_id ssh -i mykey cirros@172.16.1.37
7. Simulated kernel crash by executing "echo c > /proc/sysrq-trigger" on VM instance console and crashed the VM kernel.
8. Opened console log using "nova console-log <instance-name>" and checked for the crash message(Kernel Panic).
9. Now again tried to attach the port(demoPort2) by using "nova interface-attach --port-id <port-id> <instance-name>"

After running the above command, I didn't get any confirmation/error message on console and interface is attached successfully.
In VM details("nova show <vm-name>") I have seen the attached interface(i.e 172.16.1.36) and also the port information is not removed from the port-list( neutron port-list) .

please check the above steps and let me know if I missed any steps to reproduce the issue.

[Sreenivas (sreenivas-pothukanoori)]

Hi Michael,

I have tried to reproduce the issue in my Kilo & Liberty versions but it is working as per the expected behavior.
please find the steps which I mentioned in comment #1 and let me know your inputs on this bug to move forward.

[Sivasathurappan Radhakrishnan (siva-radhakrishnan)]

This bug lacks the necessary information to effectively reproduce and fix it, therefore it has been closed. Feel free to reopen the bug by providing the requested information and set the bug status back to ''New'