'RTNETLINK answers: Network is unreachable' error during NeutronNetworks.create_and_delete_subnets test execution

[Sergey Arkhipov]

'RTNETLINK answers: Network is unreachable' error during NeutronNetworks.create_and_delete_subnets test execution

During execution of 'NeutronNetworks.create_and_delete_subnets' Rally test scenario, I observe following stacktraces on node-12 (was observed ~2016-02-10T12:33:33)

Traceback (most recent call last):
  File /usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py, line 115, in call_driver
    getattr(driver, action)(**action_kwargs)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 204, in enable
    interface_name = self.device_manager.setup(self.network)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 1235, in setup
    self._set_default_route(network, interface_name)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py, line 1021, in _set_default_route
    device.route.add_gateway(subnet.gateway_ip)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py, line 627, in add_gateway
    self._as_root([ip_version], tuple(args))
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py, line 301, in _as_root
    use_root_namespace=use_root_namespace)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py, line 92, in _as_root
    log_fail_as_error=self.log_fail_as_error)
  File \"/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py, line 101, in _execute
    log_fail_as_error=log_fail_as_error)
  File /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py, line 159, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-6eb2fe94-ab2c-4eb8-bb2c-d2421ef8ccca', 'ip', '-4', 'route', 'replace', 'default', 'via', '1.1.1.61', 'dev', 'tap463680a6-59']
Exit code: 2
Stdin:
Stdout:
Stderr: RTNETLINK answers: Network is unreachable

Test passed without any problems.

Environment:
  10 baremetal nodes
    * 2 LMA nodes
    * 5 computes (KVM)
    * 3 controllers
      - Neutron with VLAN segmentation, no DVR and no L3 HA
      - Cinder LVM over iSCSI for volumes (Ceph is not used at all)
      - Nova quotas are not enabled

Test scenario:
{
  "NeutronNetworks.create_and_delete_subnets": [
    {
      "runner": {
        "type": "constant",
        "concurrency": 10,
        "times": 18
      },
      "args": {
        "network_create_args": {},
        "subnet_cidr_start": "1.1.0.0/30",
        "subnets_per_network": 5,
        "subnet_create_args": {}
      },
      "sla": {
        "failure_rate": {
          "max": 0
        }
      },
      "context": {
        "users": {
          "project_domain": "default",
          "users_per_tenant": 2,
          "tenants": 2,
          "resource_management_workers": 30,
          "user_domain": "default"
        },
        "quotas": {
          "neutron": {
            "subnet": -1,
            "network": -1
          }
        }
      }
    }
  ]
}

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Please find diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkOHhZNFR2R2ZMTzQ/view?usp=sharing

Log of rally execution:
https://drive.google.com/a/mirantis.com/file/d/0B9tzODpFABxkWnZxblZJVkh1VVk/view?usp=sharing

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 8.0-updates because of Medium importance. Nominated for 9.0

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Alexander Ignatov (aignatov)]

Decreased to Low since no impact at all, only errors in test logs.

[Alexander Ignatov (aignatov)]

Low bugs are moving to next-release

[Alexander Ignatov (aignatov)]

Dear Performance QA team, could you please recheck this Rally test once again in 9.0 GA build and please provide logs/snapshot if this issue occurs there.

[Denis Meltsaykin (dmeltsaykin)]

Moving to Invalid after a month in Incomplete.