12.0.10 to 13.0.1 Neutron database migration failure

[James Denton]

During an upgrade from 12.0.10 to 13.0.1, I experienced the following failure:

TASK: [os_neutron | Perform a Neutron DB online upgrade (expand)] *************
skipping: [infra02_neutron_agents_container-fb552892]
skipping: [compute01]
skipping: [compute02]
skipping: [infra03_neutron_agents_container-ad2973dd]
skipping: [infra02_neutron_server_container-45bd3fc3]
skipping: [infra03_neutron_server_container-104530c7]
failed: [infra01_neutron_server_container-96ae0d98] => {"changed": true, "cmd": ["/openstack/venvs/neutron-13.0.1/bin/neutron-db-manage", "--config-file", "/etc/neutron/neutron.conf", "--config-file", "/etc/neutron/plugins/ml2/ml2_conf.ini", "upgrade", "--expand"], "delta": "0:00:00.890617", "end": "2016-04-24 08:53:08.178402", "rc": 1, "start": "2016-04-24 08:53:07.287785", "warnings": []}
stderr: No handlers could be found for logger "oslo_config.cfg"
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/openstack/venvs/neutron-13.0.1/bin/neutron-db-manage", line 11, in <module>
    sys.exit(main())
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 749, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 223, in do_upgrade
    run_sanity_checks(config, revision)
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 731, in run_sanity_checks
    script_dir.run_env()
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/alembic/script/base.py", line 397, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
    module = load_module_py(module_id, path)
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 126, in <module>
    run_migrations_online()
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/alembic/runtime/migration.py", line 303, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 724, in check_sanity
    script.module.check_sanity(context.connection)
  File "/openstack/venvs/neutron-13.0.1/lib/python2.7/site-packages/neutron/db/migration/alembic_migrations/versions/mitaka/expand/1df244e556f5_add_unique_ha_router_agent_port_bindings.py", line 59, in check_sanity
    raise DuplicateL3HARouterAgentPortBinding(router=", ".join(res))
1df244e556f5_add_unique_ha_router_agent_port_bindings_py.DuplicateL3HARouterAgentPortBinding: Duplicate L3HARouterAgentPortBinding is created for router(s) 0b7ea3fd-dbae-43ce-a43e-6e6f89ab129e. Database cannot be upgraded. Please, remove all duplicates before upgrading the database.

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/root/setup-openstack.retry

cinder01 : ok=86 changed=10 unreachable=0 failed=0
cinder02 : ok=86 changed=9 unreachable=0 failed=0
compute01 : ok=174 changed=37 unreachable=0 failed=0
compute02 : ok=174 changed=35 unreachable=0 failed=0
infra01_cinder_api_container-ed940653 : ok=90 changed=11 unreachable=0 failed=0
infra01_cinder_scheduler_container-60e92555 : ok=78 changed=11 unreachable=0 failed=0
infra01_glance_container-b0380892 : ok=95 changed=15 unreachable=0 failed=0
infra01_keystone_container-ce0ef19a : ok=119 changed=13 unreachable=0 failed=0
infra01_neutron_agents_container-4bbf1e68 : ok=57 changed=22 unreachable=0 failed=1
infra01_neutron_server_container-96ae0d98 : ok=75 changed=27 unreachable=0 failed=1
infra01_nova_api_metadata_container-dd03b939 : ok=94 changed=17 unreachable=0 failed=0
infra01_nova_api_os_compute_container-ae3ece5b : ok=103 changed=21 unreachable=0 failed=0
infra01_nova_cert_container-db58825b : ok=94 changed=16 unreachable=0 failed=0
infra01_nova_conductor_container-9e2e5987 : ok=94 changed=16 unreachable=0 failed=0
infra01_nova_console_container-80a72af7 : ok=102 changed=18 unreachable=0 failed=0
infra01_nova_scheduler_container-4124dc10 : ok=94 changed=16 unreachable=0 failed=0
infra02_cinder_api_container-c96b6d9b : ok=78 changed=11 unreachable=0 failed=0
infra02_cinder_scheduler_container-5e7f12a4 : ok=78 changed=11 unreachable=0 failed=0
infra02_glance_container-1f4b7326 : ok=86 changed=14 unreachable=0 failed=0
infra02_keystone_container-a20285d0 : ok=104 changed=9 unreachable=0 failed=0
infra02_neutron_agents_container-fb552892 : ok=88 changed=39 unreachable=0 failed=0
infra02_neutron_server_container-45bd3fc3 : ok=72 changed=27 unreachable=0 failed=0
infra02_nova_api_metadata_container-e1836d18 : ok=94 changed=17 unreachable=0 failed=0
infra02_nova_api_os_compute_container-0055b8af : ok=94 changed=17 unreachable=0 failed=0
infra02_nova_cert_container-8cf08e27 : ok=94 changed=17 unreachable=0 failed=0
infra02_nova_conductor_container-57087fa0 : ok=94 changed=17 unreachable=0 failed=0
infra02_nova_console_container-5ff9de6f : ok=100 changed=19 unreachable=0 failed=0
infra02_nova_scheduler_container-3d2791ef : ok=94 changed=17 unreachable=0 failed=0
infra03_cinder_api_container-f29eb84f : ok=78 changed=11 unreachable=0 failed=0
infra03_cinder_scheduler_container-7661e9a2 : ok=78 changed=10 unreachable=0 failed=0
infra03_glance_container-9676f0c5 : ok=86 changed=14 unreachable=0 failed=0
infra03_keystone_container-0290dbef : ok=104 changed=9 unreachable=0 failed=0
infra03_neutron_agents_container-ad2973dd : ok=88 changed=39 unreachable=0 failed=0
infra03_neutron_server_container-104530c7 : ok=72 changed=26 unreachable=0 failed=0
infra03_nova_api_metadata_container-2bca8bc9 : ok=94 changed=16 unreachable=0 failed=0
infra03_nova_api_os_compute_container-91de7e42 : ok=94 changed=17 unreachable=0 failed=0
infra03_nova_cert_container-c55fd955 : ok=94 changed=17 unreachable=0 failed=0
infra03_nova_conductor_container-aefb0a55 : ok=94 changed=17 unreachable=0 failed=0
infra03_nova_console_container-12cedd6b : ok=100 changed=19 unreachable=0 failed=0
infra03_nova_scheduler_container-b671b195 : ok=94 changed=17 unreachable=0 failed=0

Here is the router list, including the router MyFourRouter it is complaining about:

root@infra01-utility-container-2a8d3c9a:~# neutron router-list
+--------------------------------------+--------------+------------------------------------------------------------------+-------------+------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+--------------+------------------------------------------------------------------+-------------+------+
| 0b7ea3fd-dbae-43ce-a43e-6e6f89ab129e | MyFourRouter | null | False | True |
| e127cd16-eed6-4b29-83d9-329dc0d6838f | MySixRouter | {"network_id": "cd880671-29f1-4c25-8529-c964ec682fe1", | False | True |
| | | "enable_snat": true, "external_fixed_ips": [{"subnet_id": | | |
| | | "52152718-4636-44a8-9eb3-ee3046aa7456", "ip_address": | | |
| | | "fd32:4580:9a62:801c:f816:3eff:fee8:ea16"}]} | | |
+--------------------------------------+--------------+------------------------------------------------------------------+-------------+------+

A look at the port bindings table:

MariaDB [neutron]> select * from ml2_port_bindings;
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+
| port_id | host | vif_type | vnic_type | profile | vif_details |
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+
| 023343ae-11ed-4a90-849b-cecc2fa23eec | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 18a3ee27-54a9-4440-a577-b111ef2f94bc | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 20e04c9a-caba-44ae-baaf-3185d0a43a53 | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 33a04ddb-5ede-47c0-8131-10e0e31d9209 | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 432b359b-899a-41c1-ad63-14980a75439f | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 5218ba18-415b-49c4-bab5-917abd9d9559 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| 6c5ca1f4-e5b9-4dcf-98ed-5e1c8e6d8457 | compute02 | bridge | normal | | {"port_filter": true} |
| 72a73e64-9558-49ad-9d55-7563b21ad2b5 | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 770e3b5b-000a-4bfe-8e3f-4836b674bb80 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| 79b8e7f4-a0d7-4576-866d-869b7e3dd128 | | unbound | normal | | |
| 92706bac-aa96-4cf7-801a-40fdde1371b3 | | unbound | normal | | | <-- HA port
| 93eb86ea-0ac6-45b0-a209-841ebb999ea9 | | unbound | normal | | | <-- HA port
| ab10f710-bd60-4e9e-9724-207e3dc928cd | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| be7dd04a-4ada-4daa-b005-d9aa1afb054e | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| c9a85728-fcb5-4ef8-b7ad-cc06561fd568 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| ca45f119-016f-4cd0-97fb-ec5fd231f315 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| cc1277df-8b22-4dd9-8391-1e032dab8ff8 | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| f3883c8b-7a4c-4973-8822-be526c51d510 | | unbound | normal | | | <-- HA port
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+

The ports marked as HA ports belong to the MyFourRouter router that has not yet been assigned a gateway and has not yet been scheduled as a result. Thus, the ports are unbound and no host is set. Below are the details of the ports:

root@infra01-utility-container-2a8d3c9a:~# neutron port-show 92706bac-aa96-4cf7-801a-40fdde1371b3
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| device_id | 0b7ea3fd-dbae-43ce-a43e-6e6f89ab129e |
| device_owner | network:router_ha_interface |
| dns_assignment | {"hostname": "host-169-254-192-6", "ip_address": "169.254.192.6", "fqdn": "host-169-254-192-6.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "f33da905-30f0-497d-9fb2-83bed6d69676", "ip_address": "169.254.192.6"} |
| id | 92706bac-aa96-4cf7-801a-40fdde1371b3 |
| mac_address | fa:16:3e:a1:e8:ec |
| name | HA port tenant b4d44d8a3f4844ca85613a3b2c49744a |
| network_id | b46a9f9c-3b56-4002-b03b-ac1d4d96610f |
| port_security_enabled | False |
| security_groups | |
| status | DOWN |
| tenant_id | |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
root@infra01-utility-container-2a8d3c9a:~# neutron port-show 93eb86ea-0ac6-45b0-a209-841ebb999ea9
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| device_id | 0b7ea3fd-dbae-43ce-a43e-6e6f89ab129e |
| device_owner | network:router_ha_interface |
| dns_assignment | {"hostname": "host-169-254-192-5", "ip_address": "169.254.192.5", "fqdn": "host-169-254-192-5.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "f33da905-30f0-497d-9fb2-83bed6d69676", "ip_address": "169.254.192.5"} |
| id | 93eb86ea-0ac6-45b0-a209-841ebb999ea9 |
| mac_address | fa:16:3e:4c:3a:0f |
| name | HA port tenant b4d44d8a3f4844ca85613a3b2c49744a |
| network_id | b46a9f9c-3b56-4002-b03b-ac1d4d96610f |
| port_security_enabled | False |
| security_groups | |
| status | DOWN |
| tenant_id | |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
root@infra01-utility-container-2a8d3c9a:~# neutron port-show f3883c8b-7a4c-4973-8822-be526c51d510
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| device_id | 0b7ea3fd-dbae-43ce-a43e-6e6f89ab129e |
| device_owner | network:router_ha_interface |
| dns_assignment | {"hostname": "host-169-254-192-4", "ip_address": "169.254.192.4", "fqdn": "host-169-254-192-4.openstacklocal."} |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "f33da905-30f0-497d-9fb2-83bed6d69676", "ip_address": "169.254.192.4"} |
| id | f3883c8b-7a4c-4973-8822-be526c51d510 |
| mac_address | fa:16:3e:60:96:c1 |
| name | HA port tenant b4d44d8a3f4844ca85613a3b2c49744a |
| network_id | b46a9f9c-3b56-4002-b03b-ac1d4d96610f |
| port_security_enabled | False |
| security_groups | |
| status | DOWN |
| tenant_id | |
+-----------------------+-----------------------------------------------------------------------------------------------------------------+

With the following commands, I reverted the router to a standalone router expecting Neutron to delete the HA ports:

root@infra01-utility-container-2a8d3c9a:~# neutron router-update MyFourRouter --admin-state-up=false
Updated router: MyFourRouter
root@infra01-utility-container-2a8d3c9a:~# neutron router-update MyFourRouter --ha=false
Updated router: MyFourRouter
root@infra01-utility-container-2a8d3c9a:~# neutron router-update MyFourRouter --admin-state-up=true
Updated router: MyFourRouter

MariaDB [neutron]> select * from ml2_port_bindings;
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+
| port_id | host | vif_type | vnic_type | profile | vif_details |
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+
| 023343ae-11ed-4a90-849b-cecc2fa23eec | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 18a3ee27-54a9-4440-a577-b111ef2f94bc | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 20e04c9a-caba-44ae-baaf-3185d0a43a53 | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 33a04ddb-5ede-47c0-8131-10e0e31d9209 | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| 432b359b-899a-41c1-ad63-14980a75439f | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 5218ba18-415b-49c4-bab5-917abd9d9559 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| 6c5ca1f4-e5b9-4dcf-98ed-5e1c8e6d8457 | compute02 | bridge | normal | | {"port_filter": true} |
| 72a73e64-9558-49ad-9d55-7563b21ad2b5 | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| 770e3b5b-000a-4bfe-8e3f-4836b674bb80 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| 79b8e7f4-a0d7-4576-866d-869b7e3dd128 | | unbound | normal | | |
| ab10f710-bd60-4e9e-9724-207e3dc928cd | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
| be7dd04a-4ada-4daa-b005-d9aa1afb054e | infra01_neutron_agents_container-4bbf1e68 | bridge | normal | | {"port_filter": true} |
| c9a85728-fcb5-4ef8-b7ad-cc06561fd568 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| ca45f119-016f-4cd0-97fb-ec5fd231f315 | infra02_neutron_agents_container-fb552892 | bridge | normal | | {"port_filter": true} |
| cc1277df-8b22-4dd9-8391-1e032dab8ff8 | infra03_neutron_agents_container-ad2973dd | bridge | normal | | {"port_filter": true} |
+--------------------------------------+-------------------------------------------+----------+-----------+---------+-----------------------+

After rerunning the setup-openstack playbook the operation completed successfully:

root@infra01:/opt/openstack-ansible/playbooks# openstack-ansible setup-openstack.yml

TASK: [os_neutron | Perform a Neutron DB online upgrade (expand)] *************
skipping: [infra03_neutron_agents_container-ad2973dd]
skipping: [compute02]
skipping: [infra02_neutron_agents_container-fb552892]
skipping: [compute01]
skipping: [infra03_neutron_server_container-104530c7]
skipping: [infra02_neutron_server_container-45bd3fc3]
changed: [infra01_neutron_server_container-96ae0d98]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/312274

[Jesse Pretorius (jesse-pretorius)]

Resetting triage status as it is not clear from the comment log that anyone has actually confirmed that this is a repeatable issue. The bug status was changed by the OpenStack-CI bot. The reset of this to 'New' will highlight this bug back into the triage process.

[Launchpad Janitor (janitor)]

[Expired for openstack-ansible because there has been no activity for 60 days.]