Tests are leaving network resources behind

[Salvatore Orlando]

After the changes for selectively creating resources were introduced, tempest is leaving network resources behind.
This also implies the possibility of having a consistent number of active dnsmasq and neutron-ns-metadata-proxy active processes, which could be a possibility of unexpected gate failures.

This is not a duplicate of bug 1157306

[Salvatore Orlando (salvatore-orlando)]

I don't know if this is related to the kernel crash, but at this stage I would try everything :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/70169

[Salvatore Orlando (salvatore-orlando)]

The kernel bug manifests when mounting of nbd devices is combined with ip namespace operations.
Using openstack it can be reproduced only with the following configuration:
- compute service must run on the same node as the dhcp-agent and/or the l3-agent
- file injection should be turned on: libvirt.inject_partition != -2
- key injection should be enabled: libvirt.inject_key = True
- config drive should be disabled: force_config_drive = 'False' or empty string or None

If these conditions are met, nbd mount will be used to inject the key into the instance. This will trigger something in the kernel which subsequently will cause the crash in a process running in a network namespace. After this crash nbd mount won't work anymore.

Using openstack the crash can be reproduced within a few minutes with the scripts available here: https://gist.github.com/salv-orlando/8715991

- keep_booting_stuff.sh creates and destroys vms continuously, ensuring a key is always injected
- stress_me_to_death creates network namespaces, launches a process in them, kills the process and then the namespace

The two scripts require openstack and should be executed concurrently.
Crashdump data will posted as soon as possible.

It should not be too hard to provide a script that reproduces the issue and is indepedent from openstack.

[Salvatore Orlando (salvatore-orlando)]

sorry, wrong bug!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70169
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=cf996c653bef62f2ae71331cf45323e5f183b2be
Submitter: Jenkins
Branch: master

commit cf996c653bef62f2ae71331cf45323e5f183b2be
Author: Salvatore Orlando <email address hidden>
Date: Thu Jan 30 09:15:18 2014 -0800

    Remove network resources created in scenario tests

    Change-Id: I1bd53d99f2632441ebf648cb0c2929c9b55998b6
    Closes-Bug: #1274410