Live migration does not work anymore in latest version of sunbeam
Reproducer:
sunbeam cloud-config -u
sunbeam cloud-config -u -a
sunbeam launch --name demo-server
export OS_CLOUD=sunbeam-admin
openstack server migrate --block-migration --live-migration b6ed09e3-920a-4f3b-9842-20b29ace5c81
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'nova.exception_Remote.MigrationError_Remote'> (HTTP 500) (Request-ID: req-868029f0-52c6-4d99-8aeb-4b28f25e2758)
2024-06-04T15:15:21.651Z [wsgi-nova-api] 2024-06-04 15:15:21.651500 2024-06-04 15:15:21.650 28 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer\x1b[00m
2024-06-04T15:15:22.792Z [wsgi-nova-api] 2024-06-04 15:15:22.791327 2024-06-04 15:15:22.790 28 INFO nova.api.openstack.requestlog [None req-84c82319-785a-4cb8-a806-1229be269c1a 23eeb78ce62645b59aa768042ec6d2f4 437766bf79134ca4b79daf5fcc36ad52 - - 780c5458df8b46469081d08eff41c3c5 780c5458df8b46469081d08eff41c3c5] 10.1.77.82 "GET /openstack-nova/v2.1/servers/b6ed09e3-920a-4f3b-9842-20b29ace5c81" status: 200 len: 2147 microversion: 2.91 time: 1.142780\x1b[00m
2024-06-04T15:15:22.793Z [wsgi-nova-api] 10.1.77.82 - - [04/Jun/2024:15:15:21 +0000] "GET /openstack-nova/v2.1/servers/b6ed09e3-920a-4f3b-9842-20b29ace5c81 HTTP/1.1" 200 2550 "-" "openstacksdk/3.0.0 keystoneauth1/5.6.0 python-requests/2.31.0 CPython/3.10.12"
2024-06-04T15:15:25.779Z [wsgi-nova-api] 2024-06-04 15:15:25.779464 2024-06-04 15:15:25.779 26 INFO nova.api.openstack.requestlog [req-a4b56dd1-bff9-4de6-9073-6a45cdcbff4d req-8a0f0bbb-dad8-4a96-83e3-b3fb4fc9e136 4daf15884e6943afb17ef98575edce11 a83b877de55449438f4df90fd964c937 - - 801c6aa3f06a4e2a8f11c225423c6dff 801c6aa3f06a4e2a8f11c225423c6dff] ::1 "GET /" status: 200 len: 367 microversion: - time: 0.000451\x1b[00m
2024-06-04T15:15:25.779Z [wsgi-nova-api] ::1 - - [04/Jun/2024:15:15:25 +0000] "GET / HTTP/1.1" 200 508 "-" "Go-http-client/1.1"
2024-06-04T15:15:32.978Z [pebble] GET /v1/notices?timeout=30s 30.002077743s 200
2024-06-04T15:15:35.780Z [wsgi-nova-api] 2024-06-04 15:15:35.780576 2024-06-04 15:15:35.780 29 INFO nova.api.openstack.requestlog [req-21c58828-5305-49e0-9439-ea095f157829 req-da609c72-0540-4e03-8ff4-b7b7ea482df1 4daf15884e6943afb17ef98575edce11 a83b877de55449438f4df90fd964c937 - - 801c6aa3f06a4e2a8f11c225423c6dff 801c6aa3f06a4e2a8f11c225423c6dff] ::1 "GET /" status: 200 len: 367 microversion: - time: 0.000587\x1b[00m
2024-06-04T15:15:35.781Z [wsgi-nova-api] ::1 - - [04/Jun/2024:15:15:35 +0000] "GET / HTTP/1.1" 200 508 "-" "Go-http-client/1.1"
2024-06-04T15:15:38.096Z [wsgi-nova-api] 2024-06-04 15:15:38.096666 2024-06-04 15:15:38.096 27 INFO nova.api.openstack.requestlog [None req-640be386-70b4-4d6e-a8b1-8c782dca029f 23eeb78ce62645b59aa768042ec6d2f4 437766bf79134ca4b79daf5fcc36ad52 - - 780c5458df8b46469081d08eff41c3c5 780c5458df8b46469081d08eff41c3c5] 10.1.77.82 "GET /openstack-nova/v2.1/servers/b6ed09e3-920a-4f3b-9842-20b29ace5c81" status: 200 len: 1788 microversion: 2.1 time: 1.405721\x1b[00m
2024-06-04T15:15:38.097Z [wsgi-nova-api] 10.1.77.82 - - [04/Jun/2024:15:15:36 +0000] "GET /openstack-nova/v2.1/servers/b6ed09e3-920a-4f3b-9842-20b29ace5c81 HTTP/1.1" 200 2189 "-" "python-novaclient"
2024-06-04T15:15:45.780Z [wsgi-nova-api] 2024-06-04 15:15:45.780528 2024-06-04 15:15:45.780 26 INFO nova.api.openstack.requestlog [req-a4b56dd1-bff9-4de6-9073-6a45cdcbff4d req-8a0f0bbb-dad8-4a96-83e3-b3fb4fc9e136 4daf15884e6943afb17ef98575edce11 a83b877de55449438f4df90fd964c937 - - 801c6aa3f06a4e2a8f11c225423c6dff 801c6aa3f06a4e2a8f11c225423c6dff] ::1 "GET /" status: 200 len: 367 microversion: - time: 0.000700\x1b[00m
2024-06-04T15:15:45.780Z [wsgi-nova-api] ::1 - - [04/Jun/2024:15:15:45 +0000] "GET / HTTP/1.1" 200 508 "-" "Go-http-client/1.1"
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323305 2024-06-04 15:15:47.311 28 ERROR nova.api.openstack.wsgi [None req-868029f0-52c6-4d99-8aeb-4b28f25e2758 23eeb78ce62645b59aa768042ec6d2f4 437766bf79134ca4b79daf5fcc36ad52 - - 780c5458df8b46469081d08eff41c3c5 780c5458df8b46469081d08eff41c3c5] Unexpected exception in API method: nova.exception_Remote.MigrationError_Remote: Migration error: rule:create_port_binding is disallowed by policy
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323322 Neutron server returns request_ids: ['req-deeb8a4c-bfc2-4611-b4a8-fb18d8c48896']
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323324 Traceback (most recent call last):
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323324
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323325 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323326 ret = obj(*args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323326
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323327 File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 272, in _handle_fault_response
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323328 exception_handler_v20(status_code, error_body)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323329
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323330 File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 90, in exception_handler_v20
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323331 raise client_exc(message=error_message,
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323332
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323332 neutronclient.common.exceptions.Forbidden: rule:create_port_binding is disallowed by policy
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323333 Neutron server returns request_ids: ['req-deeb8a4c-bfc2-4611-b4a8-fb18d8c48896']
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323333
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323334
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323334 During handling of the above exception, another exception occurred:
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323335
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323335
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323336 Traceback (most recent call last):
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323336
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323337 File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 505, in _live_migrate
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323337 task.execute()
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323338
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323338 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/base.py", line 25, in wrap
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323339 with excutils.save_and_reraise_exception():
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323339
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323340 File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323340 self.force_reraise()
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323341
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323341 File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323342 raise self.value
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323342
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323343 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/base.py", line 23, in wrap
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323343 return original(self)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323344
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323344 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/base.py", line 40, in execute
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323345 return self._execute()
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323345
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323346 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/live_migrate.py", line 100, in _execute
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323346 self.destination, dest_node, self.limits = self._find_destination()
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323347
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323347 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/live_migrate.py", line 550, in _find_destination
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323348 self._call_livem_checks_on_host(host, provider_mapping)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323348
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323349 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/live_migrate.py", line 370, in _call_livem_checks_on_host
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323349 bindings = self._bind_ports_on_destination(
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323350
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323350 File "/usr/lib/python3/dist-packages/nova/conductor/tasks/live_migrate.py", line 411, in _bind_ports_on_destination
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323351 bindings = self.network_api.bind_ports_to_host(
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323352
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323352 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 1541, in bind_ports_to_host
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323354 binding = client.create_port_binding(port_id, data)['binding']
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323355
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323361 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323362 ret = obj(*args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323363
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323363 File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 837, in create_port_binding
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323364 return self.post(self.port_bindings_path % port_id, body=body)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323364
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323365 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323365 ret = obj(*args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323366
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323366 File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 361, in post
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323367 return self.do_request("POST", action, body=body,
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323367
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323368 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 196, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323368 ret = obj(*args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323369
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323369 File "/usr/lib/python3/dist-packages/neutronclient/v2_0/client.py", line 297, in do_request
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323370 self._handle_fault_response(status_code, replybody, resp)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323371
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323371 File "/usr/lib/python3/dist-packages/nova/network/neutron.py", line 214, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323372 raise exception.Forbidden(str(e))
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323372
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323373 nova.exception.Forbidden: rule:create_port_binding is disallowed by policy
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323373 Neutron server returns request_ids: ['req-deeb8a4c-bfc2-4611-b4a8-fb18d8c48896']
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323374
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323374
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323374 During handling of the above exception, another exception occurred:
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323375
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323375
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323376 Traceback (most recent call last):
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323376
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323377 File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323377 res = self.dispatcher.dispatch(message)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323378
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323378 File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323379 return self._do_dispatch(endpoint, method, ctxt, args)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323379
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323380 File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323381 result = func(ctxt, **new_args)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323381
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323382 File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 244, in inner
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323382 return func(*args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323383
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323383 File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 102, in wrapper
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323384 return fn(self, context, *args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323384
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323385 File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1453, in decorated_function
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323385 return function(self, context, *args, **kwargs)
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323386
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323386 File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 327, in migrate_server
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323387 self._live_migrate(context, instance, scheduler_hint,
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323387
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323388 File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 534, in _live_migrate
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323388 raise exception.MigrationError(reason=str(ex))
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323389
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323389 nova.exception.MigrationError: Migration error: rule:create_port_binding is disallowed by policy
2024-06-04T15:15:47.323Z [wsgi-nova-api] 2024-06-04 15:15:47.323390 Neutron server returns request_ids: ['req-deeb8a4c-bfc2-4611-b4a8-fb18d8c48896']
This is due to 2 changes: /review. opendev. org/c/openstack /keystone/ +/863420 <-- adding a service role /review. opendev. org/c/openstack /neutron/ +/884613 <-- adding a policy create_port_binding only allowing service role
- https:/
- https:/
Adding service role to nova's user solves this issue.