NeutronNetworks.create_and_delete_subnets fails NotFound

[Bas de Bruijne]

Rally test NeutronNetworks.create_and_delete_subnets fails on 7th iteration.

-----------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally/task/runner.py", line 69, in _run_scenario_once
    getattr(scenario_inst, method_name)(**scenario_kwargs)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally_openstack/task/scenarios/neutron/network.py", line 233, in run
    subnet_cidr_start, subnets_per_network)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally_openstack/task/scenarios/neutron/utils.py", line 312, in _create_subnets
    for i in range(subnets_per_network)]
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally_openstack/task/scenarios/neutron/utils.py", line 312, in <listcomp>
    for i in range(subnets_per_network)]
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally_openstack/task/scenarios/neutron/utils.py", line 146, in _create_subnet
    return {"subnet": self.neutron.create_subnet(**subnet_create_args)}
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally/task/service.py", line 114, in wrapper
    return func(instance, *args, **kwargs)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally/task/atomic.py", line 91, in func_atomic_actions
    f = func(self, *args, **kwargs)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/rally_openstack/common/services/network/neutron.py", line 489, in create_subnet
    subnet = self.client.create_subnet({"subnet": body})["subnet"]
  File "/snap/fcbtest/14/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 873, in create_subnet
    return self.post(self.subnets_path, body=body)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 359, in post
    headers=headers, params=params)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 294, in do_request
    self._handle_fault_response(status_code, replybody, resp)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 269, in _handle_fault_response
    exception_handler_v20(status_code, error_body)
  File "/snap/fcbtest/14/lib/python3.6/site-packages/neutronclient/v2_0/client.py", line 93, in exception_handler_v20
    request_ids=request_ids)
neutronclient.common.exceptions.NotFound: Subnet c48d9cfd-edec-4691-a1e2-b0f169cfbde5 could not be found.
Neutron server returns request_ids: ['req-96c99100-069f-40c2-aa2e-e4bc5b860bae']
-----------------------------------------------------------------------------------------------

neutron-server.service keeps failing and restarting:

-----------------------------------------------------------------------------------------------

Sep 1 14:06:06 juju-74c5d5-1-lxd-6 neutron-server[8435]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Failed with result 'exit-code'.
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Service hold-off time over, scheduling restart.
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Scheduled restart job, restart counter is at 6.
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: Stopped OpenStack Neutron Server.
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Failed to reset devices.list: Operation not permitted
Sep 1 14:06:07 juju-74c5d5-1-lxd-6 systemd[1]: Started OpenStack Neutron Server.
Sep 1 14:06:10 juju-74c5d5-1-lxd-6 neutron-server[8598]: /usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
Sep 1 14:06:10 juju-74c5d5-1-lxd-6 neutron-server[8598]: return pkg_resources.EntryPoint.parse("x=" + s).load(False)
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Main process exited, code=exited, status=1/FAILURE
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Failed with result 'exit-code'.
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Service hold-off time over, scheduling restart.
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Scheduled restart job, restart counter is at 7.
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: Stopped OpenStack Neutron Server.
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: neutron-server.service: Failed to reset devices.list: Operation not permitted
Sep 1 14:06:11 juju-74c5d5-1-lxd-6 systemd[1]: Started OpenStack Neutron Server.
Sep 1 14:06:15 juju-74c5d5-1-lxd-6 neutron-server[8695]: /usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py:22: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.

-----------------------------------------------------------------------------------------------

logs:
https://oil-jenkins.canonical.com/artifacts/09956e7b-46e6-48c9-9631-dd0b84c4a808/generated/generated/openstack/rally/auto_generated.html

https://oil-jenkins.canonical.com/artifacts/09956e7b-46e6-48c9-9631-dd0b84c4a808/generated/generated/openstack/juju-crashdump-openstack-2021-09-01-16.40.40.tar.gz

[Brian Haley (brian-haley)]

Have you debugged why the neutron-server container is generating this error:

  Failed to reset devices.list: Operation not permitted

That and the pkg_resources look suspicious.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Same comment as Brian: you should first debug the issue with this not permitted operation. What version are you using?

I don't see any Rally problem in master or stable branches.

BTW, I can't access to those links. Message: "Either you have not been granted access to this resource or your entitlement has timed out. Please try again."

[Bas de Bruijne (basdbruijne)]

Bug has been moved to neutron-charm to determine if it's a downstream error.

https://bugs.launchpad.net/charm-neutron-api/+bug/1942608

I will look into the operation not permitted message. This is from an automated test and the machines are not kept alive after they fail, so debugging is not easy.

@Radolfo, you can find the machine log here https://drive.google.com/file/d/17c43eNDUUK7YsINIi0AvHzJ-GlG1ovvM/view?usp=sharing

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Bas:

I reviewed the Neutron logs and the problem is here, just before the failing call "req-96c99100-069f-40c2-aa2e-e4bc5b860bae":
2021-09-01 15:48:15.712 150762 WARNING neutron.scheduler.dhcp_agent_scheduler [req-50c14045-64d1-42d0-b007-6d5f5765c829 b821889f2846473fb33c3e81939200fa cc8b1138f7eb4ab5a8a2ca827c7f8868 - 4ac8c9e3c2e24e1f84a4e600fc641f0d 4ac8c9e3c2e24e1f84a4e600fc641f0d] No more DHCP agents

I don't know how are you deploying or installing Neutron (using juju, I guess). You are deploying the DHCP service but this is not needed in OVN. You should disable it before starting the Neutron server.

Actually, you don't need any RPC worker running (as long as you don't have L3 or DHCP agents and you don't need SR-IOV).

Regards.

[Konstantinos Kaskavelis (kaskavel)]

Closing this due to inactivity (low number of occurrences, and no hit for more than one year)