neutron_ovn_db_sync_util error No more IP addresses available

[Jake Hill]

Migrating charmed openstack (focal/ussuri) from OVS to OVN, following the instructions at;

  https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-ovn.html#migration-from-neutron-ml2-ovs-to-ml2-ovn

The data migration step is failing (see error below). This seems to result in a partial migration. The neutron-api units are paused so it is not possible to diagnose the offending network. I'm unclear as to if I can resume the API safely at this point.

$ juju run-action --wait neutron-api-plugin-ovn/0 migrate-ovn-db i-really-mean-it=true

[snip]

    Stdout: "migrate-ovn-db: OUTPUT FROM SYNC ON STDOUT:\n2020-11-25 11:06:17.760
      413761 INFO neutron.cmd.ovn.neutron_ovn_db_sync_util [-] Started Neutron OVN
      db sync\e[00m\n2020-11-25 11:06:18.140 413761 INFO neutron.manager [-] Loading
      core plugin: neutron.cmd.ovn.neutron_ovn_db_sync_util.Ml2Plugin\e[00m\n

[snip]

      11:06:57.227 413761 INFO neutron.db.ovn_revision_numbers_db [req-91be95af-c8c7-41cc-a372-5cd7a0241373
      - - - - -] Successfully bumped revision number for resource e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575
      (type: networks) to 10\e[00m\n2020-11-25 11:06:57.565 413761 CRITICAL neutron_ovn_db_sync_util
      [req-91be95af-c8c7-41cc-a372-5cd7a0241373 - - - - -] Unhandled error: neutron_lib.exceptions.IpAddressGenerationFailure:
      No more IP addresses available on network e50ff198-0fb5-4fd4-9b6c-d2f4b0a3f575.\n2020-11-25
      11:06:57.565 413761 ERROR neutron_ovn_db_sync_util Traceback (most recent call
      last):\n2020-11-25 11:06:57.565 413761 ERROR neutron_ovn_db_sync_util File
      \"/usr/lib/python3/dist-packages/neutron/db/ipam_pluggable_backend.py\", line
      138, in _ipam_allocate_ips\n2020-11-25 11:06:57.565 413761 ERROR neutron_ovn_db_sync_util
      \ ip_address, subnet_id = ipam_allocator.allocate(ip_request)\n2020-11-25
      11:06:57.565 413761 ERROR neutron_ovn_db_sync_util File \"/usr/lib/python3/dist-packages/neutron/ipam/subnet_alloc.py\",
      line 240, in allocate\n2020-11-25 11:06:57.565 413761 ERROR neutron_ovn_db_sync_util
      \ raise ipam_exc.IpAddressGenerationFailureAllSubnets()\n2020-11-25 11:06:57.565
      413761 ERROR neutron_ovn_db_sync_util neutron.ipam.exceptions.IpAddressGenerationFailureAllSubnets:
      No more IP addresses available.\n2020-11-25 11:06:57.565 413761 ERROR neutron_ovn_db_sync_util
      \n2020-11-25 11:06:57.565 413761 ERROR neutron_ovn_db_sync_util During handling
      of the above exception, another exception occurred:\n

[Jake Hill (routergod)]

Sorry this is not your bug. I raised https://bugs.launchpad.net/networking-ovn/+bug/1905566

[Jake Hill (routergod)]

Oh what a mess! Please ignore that previous comment, it was intended for a different report. This *is* indeed where I intended to file this!

[Jake Hill (routergod)]

Meanwhile I found a clue here;

  https://bugs.launchpad.net/charm-neutron-api-plugin-ovn/+bug/1905554