sunbeam configure fails with Error creating openstack_networking_subnet_v2

[Marian Gasparovic]

2023-09-01-22:46:35 root DEBUG openstack_images_image_v2.ubuntu: Still creating... [30s elapsed]
2023-09-01-22:46:35 root DEBUG openstack_images_image_v2.ubuntu: Creation complete after 39s [id=9f1743ac-d5f3-4041-b476-0b59d63c17d6]
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG Error: Error creating openstack_networking_subnet_v2: Internal Server Error
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG with openstack_networking_subnet_v2.external_subnet,
2023-09-01-22:46:35 root DEBUG on main.tf line 71, in resource "openstack_networking_subnet_v2" "external_subnet":
2023-09-01-22:46:35 root DEBUG 71: resource "openstack_networking_subnet_v2" "external_subnet" {
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG Error configuring cloud
2023-09-01-22:46:35 root DEBUG Traceback (most recent call last):
2023-09-01-22:46:35 root DEBUG File "/snap/openstack/254/lib/python3.10/site-packages/sunbeam/commands/terraform.py", line 200, in apply
2023-09-01-22:46:35 root DEBUG process = subprocess.run(
2023-09-01-22:46:35 root DEBUG File "/usr/lib/python3.10/subprocess.py", line 526, in run
2023-09-01-22:46:35 root DEBUG raise CalledProcessError(retcode, process.args,
2023-09-01-22:46:35 root DEBUG subprocess.CalledProcessError: Command '['/snap/openstack/254/bin/terraform', 'apply', '-auto-approve', '-no-color']' returned non-zero exit status 1.
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG During handling of the above exception, another exception occurred:
2023-09-01-22:46:35 root DEBUG
2023-09-01-22:46:35 root DEBUG Traceback (most recent call last):
2023-09-01-22:46:35 root DEBUG File "/snap/openstack/254/lib/python3.10/site-packages/sunbeam/commands/configure.py", line 585, in run
2023-09-01-22:46:35 root DEBUG self.tfhelper.apply()
2023-09-01-22:46:35 root DEBUG File "/snap/openstack/254/lib/python3.10/site-packages/sunbeam/commands/terraform.py", line 214, in apply
2023-09-01-22:46:35 root DEBUG raise TerraformException(str(e))
2023-09-01-22:46:35 root DEBUG sunbeam.commands.terraform.TerraformException: Command '['/snap/openstack/254/bin/terraform', 'apply', '-auto-approve', '-no-color']' returned non-zero exit status 1.
2023-09-01-22:46:35 root DEBUG Error: Command '['/snap/openstack/254/bin/terraform', 'apply', '-auto-approve', '-no-color']' returned non-zero exit status 1.

logs - https://oil-jenkins.canonical.com/artifacts/85ba9cf2-9720-420a-b8f0-21e78804b22a/index.html

[James Page (james-page)]

2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation [None req-06248b5f-c9ee-482c-b51b-69ad534b9bb9 17da3875f2b94038b78753c558540f5e d55b9a608b664a78822ccffeb969ca41 - - f520625b9a944bdd8362f8432af29196 f520625b9a944bdd8362f8432af29196] POST failed.: TypeError: 'NoneType' object is not subscriptable
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/pecan/core.py", line 693, in __call__
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/pecan/core.py", line 584, in invoke_controller
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 140, in wrapped
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception():
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation raise self.value
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/neutron_lib/db/api.py", line 138, in wrapped
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/oslo_db/api.py", line 144, in wrapper
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation with excutils.save_and_reraise_exception() as ectxt:
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:04.432 44 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-01T22:46:04.440Z [neutron-server] 2023-09-01 22:46:0...

[James Page (james-page)]

The ordering of the creation of openstack objects by the terraform plan is exposing what I think is a bug in neutron - the use router is created before the external subnet for the external network - when the subnet gets created, it has post creation code that should go and update all of the routers attached to the external network - this is where the exception that causes the internal server error is being thrown from.

So depending of speed and ordering this may or may not get triggered.

neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/neutron/db/db_base_plugin_v2.py", line 766, in _create_subnet_postcommit
neutron.pecan_wsgi.hooks.translation self._update_router_gw_ports(context,
neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/neutron/db/db_base_plugin_v2.py", line 726, in _update_router_gw_ports
neutron.pecan_wsgi.hooks.translation self._update_router_gw_port(context, id, network, subnet)
neutron.pecan_wsgi.hooks.translation File "/usr/lib/python3/dist-packages/neutron/db/db_base_plugin_v2.py", line 739, in _update_router_gw_port
neutron.pecan_wsgi.hooks.translation fips = [f for f in external_gateway_info['external_fixed_ips']
neutron.pecan_wsgi.hooks.translation TypeError: 'NoneType' object is not subscriptable

[James Page (james-page)]

Fix in candidate channel