Neutron DHCP Agent KeyError during NeutronNetworks.create_and_delete_subnets test execution

[Sergey Arkhipov]

During execution of 'NeutronNetworks.create_and_delete_subnets' Rally test scenario, I observe following stacktraces on node-13 (was observed ~2016-02-10T12:33:34). MOS 8.0 RC1 (build 529) is used.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 115, in call_driver
    getattr(driver, action)(**action_kwargs)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 206, in enable
    self.spawn_process()
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 414, in spawn_process
    self._spawn_or_reload_process(reload_with_HUP=False)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 423, in _spawn_or_reload_process
    self._output_config_files()
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 450, in _output_config_files
    self._output_opts_file()
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 746, in _output_opts_file
    options, subnet_index_map = self._generate_opts_per_subnet()
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 796, in _generate_opts_per_subnet
    subnet_dhcp_ip = subnet_to_interface_ip[subnet.id]
KeyError: u'0b2f1504-9a27-4919-ba6d-6065c8b23432'

Test passed without any problems.

Environment:
  10 baremetal nodes
    * 2 LMA nodes
    * 5 computes (KVM)
    * 3 controllers
      - Neutron with VLAN segmentation, no DVR and no L3 HA
      - Cinder LVM over iSCSI for volumes (Ceph is not used at all)
      - Nova quotas are not enabled

Test scenario:
{
  "NeutronNetworks.create_and_delete_subnets": [
    {
      "runner": {
        "type": "constant",
        "concurrency": 10,
        "times": 18
      },
      "args": {
        "network_create_args": {},
        "subnet_cidr_start": "1.1.0.0/30",
        "subnets_per_network": 5,
        "subnet_create_args": {}
      },
      "sla": {
        "failure_rate": {
          "max": 0
        }
      },
      "context": {
        "users": {
          "project_domain": "default",
          "users_per_tenant": 2,
          "tenants": 2,
          "resource_management_workers": 30,
          "user_domain": "default"
        },
        "quotas": {
          "neutron": {
            "subnet": -1,
            "network": -1
          }
        }
      }
    }
  ]
}

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Please find diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkOHhZNFR2R2ZMTzQ/view?usp=sharing

Log of rally execution:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkWnZxblZJVkh1VVk/view?usp=sharing

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 8.0-updates because of Medium importance. Nominated for 9.0

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Ignatov (aignatov)]

Test passed without any problems, decreasing to Low importance.

[Alexander Ignatov (aignatov)]

Dear Performance QA team, could you please recheck this Rally test once again in 9.0 GA build and please provide logs/snapshot if this issue occurs there.

[Denis Meltsaykin (dmeltsaykin)]

Moving to Invalid after a month in Incomplete.