DB migration is broken with two unassigned floating IPs

[Dr. Jens Harbott]

The error looks like this:

STDERR: INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
Traceback (most recent call last):
  File "/usr/bin/neutron-db-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 686, in main
    return_val |= bool(CONF.command.func(config, CONF.command.name))
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 205, in do_upgrade
    run_sanity_checks(config, revision)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 670, in run_sanity_checks
    script_dir.run_env()
  File "/usr/lib/python2.7/dist-packages/alembic/script/base.py", line 407, in run_env
    util.load_python_file(self.dir, 'env.py')
  File "/usr/lib/python2.7/dist-packages/alembic/util/pyfiles.py", line 93, in load_python_file
    module = load_module_py(module_id, path)
  File "/usr/lib/python2.7/dist-packages/alembic/util/compat.py", line 79, in load_module_py
    mod = imp.load_source(module_id, path, fp)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 120, in <module>
    run_migrations_online()
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/env.py", line 114, in run_migrations_online
    context.run_migrations()
  File "<string>", line 8, in run_migrations
  File "/usr/lib/python2.7/dist-packages/alembic/runtime/environment.py", line 797, in run_migrations
    self.get_context().run_migrations(**kw)
  File "/usr/lib/python2.7/dist-packages/alembic/runtime/migration.py", line 303, in run_migrations
    for step in self._migrations_fn(heads, self):
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/cli.py", line 663, in check_sanity
    script.module.check_sanity(context.connection)
  File "/usr/lib/python2.7/dist-packages/neutron/db/migration/alembic_migrations/versions/newton/expand/6b461a21bcfc_uniq_floatingips0floating_network_.py", line 59, in check_sanity
    raise DuplicateFloatingIPforOneFixedIP(fixed_ip_address=",".join(res))
TypeError: sequence item 0: expected string, NoneType found

with the following database entries:
mysql> select * from floatingips;
+----------------------------------+--------------------------------------+---------------------+--------------------------------------+--------------------------------------+--------------------------------------+------------------+--------------------------------------+--------------------------------------+--------+------------------+
| tenant_id | id | floating_ip_address | floating_network_id | floating_port_id | fixed_port_id | fixed_ip_address | router_id | last_known_router_id | status | standard_attr_id |
+----------------------------------+--------------------------------------+---------------------+--------------------------------------+--------------------------------------+--------------------------------------+------------------+--------------------------------------+--------------------------------------+--------+------------------+
| f0081f0762a443cbbad09b186557721b | 52c6641f-fcd9-4d7a-bcab-54739353b122 | xxx.29 | 768ae947-9346-4373-b9bf-ebf6c82a7187 | f4c857a1-27ca-4a59-931b-379569eb8f72 | NULL | NULL | NULL | NULL | DOWN | 132 |
| f0081f0762a443cbbad09b186557721b | 57abe18d-d886-4532-b82f-40e93b07b1d7 | xxx.52 | 768ae947-9346-4373-b9bf-ebf6c82a7187 | 54e44a59-13b1-4926-ac67-8c90e7dd41a6 | 3f862ab3-ea76-4b52-8618-7ce2f5a9b13d | 10.11.0.30 | 3865b2e8-0764-4084-a565-db1b790011a8 | NULL | ACTIVE | 3062 |
| f0081f0762a443cbbad09b186557721b | 67c32788-5d79-410f-9c6a-de6caa40cf29 | xxx.31 | 768ae947-9346-4373-b9bf-ebf6c82a7187 | 7bba70b8-3e56-4854-9af5-38efd0703597 | 59365d06-13e2-4c99-b167-1455caa032ca | 10.11.0.34 | 3865b2e8-0764-4084-a565-db1b790011a8 | 3865b2e8-0764-4084-a565-db1b790011a8 | ACTIVE | 183 |
| f0081f0762a443cbbad09b186557721b | 85043a0c-e294-4d61-a54f-2e5171898c70 | xxx.53 | 768ae947-9346-4373-b9bf-ebf6c82a7187 | fa689c0f-168a-4a86-8011-870711ebcaf5 | NULL | NULL | NULL | NULL | DOWN | 3089 |
| f0081f0762a443cbbad09b186557721b | 88f11e58-cd78-4c6d-a2a6-b35aa32c7a00 | xxx.51 | 768ae947-9346-4373-b9bf-ebf6c82a7187 | 58a7f431-0d78-4916-ac73-b8592dfea018 | ef7ec32a-b968-4d73-9f91-24a441e9b59a | 10.11.0.31 | 3865b2e8-0764-4084-a565-db1b790011a8 | NULL | ACTIVE | 3054 |
+----------------------------------+--------------------------------------+---------------------+--------------------------------------+--------------------------------------+--------------------------------------+------------------+--------------------------------------+--------------------------------------+--------+------------------+
5 rows in set (0.00 sec)

This happens on upgrade from Mitaka to Newton rc1.

[Dr. Jens Harbott (j-harbott)]

O.k., seems my initial idea was correct, if I manually delete one of the rows with fixed_ip_address==NULL, the error disappears.

[Henry Gessau (gessau)]

It's supposed to show you this message:
https://github.com/openstack/neutron/blob/9.0.0.0rc2/neutron/db/migration/alembic_migrations/versions/newton/expand/6b461a21bcfc_uniq_floatingips0floating_network_.py#L43-L46

Maybe we need to make the sanity check failure messages show in a better way.

[Dr. Jens Harbott (j-harbott)]

@Henry:

1. Two floating IPs being unassigned, i.e. having fixed_ip=NULL, does not count as duplicate for me and you will break lots of existing deployments if you change that.
2. If you look closely, you will see that it fails to show the message because ",".join(NULL) raises a Type Error.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/378938

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/379064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/378938
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a80b89b6fe1611a68d34684d9e80ad606f115366
Submitter: Jenkins
Branch: master

commit a80b89b6fe1611a68d34684d9e80ad606f115366
Author: Henry Gessau <email address hidden>
Date: Wed Sep 28 13:33:13 2016 -0400

    Ignore NULL fixed_ips for duplicate floating_ip check

    Closes-Bug: #1628549

    Change-Id: Ie49b46af93697ca67009373eadce9ba4f312dcc0

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/379064
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=71f2d2bc90aaf9ee696cd4e4f29879ba6c5703b5
Submitter: Jenkins
Branch: stable/newton

commit 71f2d2bc90aaf9ee696cd4e4f29879ba6c5703b5
Author: Henry Gessau <email address hidden>
Date: Wed Sep 28 13:33:13 2016 -0400

    Ignore NULL fixed_ips for duplicate floating_ip check

    Closes-Bug: #1628549

    (cherry picked from commit a80b89b6fe1611a68d34684d9e80ad606f115366)
    Change-Id: Ie49b46af93697ca67009373eadce9ba4f312dcc0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0rc3 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.0.0b1 development milestone.