Functional db migration tests broken

[Slawek Kaplonski]

It seems that it's failing all the time now. Example of failure https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ba6/790999/7/check/neutron-functional-with-uwsgi/ba6f15c/testr_results.html

Stacktrace:

ft1.4: neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql.test_models_synctesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 125, in inner
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/db/test_migrations.py", line 385, in test_models_sync
    super(TestModelsMigrationsMysql, self).test_models_sync()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/test_migrations.py", line 597, in test_models_sync
    self.fail(
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Models and migration scripts aren't in sync:
[ ( 'add_index',
    Index('ix_address_groups_project_id', Column('project_id', String(length=255), table=<address_groups>))),
  ( 'add_index',
    Index('ix_address_scopes_project_id', Column('project_id', String(length=255), table=<address_scopes>))),
  ( 'add_index',
    Index('ix_addressgrouprbacs_project_id', Column('project_id', String(length=255), table=<addressgrouprbacs>))),
  ( 'add_constraint',
    UniqueConstraint(Column('mac_address', NullType(), table=<dvr_host_macs>))),
  ( 'add_index',
    Index('ix_floatingipdnses_floatingip_id', Column('floatingip_id', String(length=36), table=<floatingipdnses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_floatingips_project_id', Column('project_id', String(length=255), table=<floatingips>))),
  ( 'add_index',
    Index('ix_logs_project_id', Column('project_id', String(length=255), table=<logs>))),
  ( 'add_index',
    Index('ix_logs_resource_id', Column('resource_id', String(length=36), table=<logs>))),
  ( 'add_index',
    Index('ix_logs_target_id', Column('target_id', String(length=36), table=<logs>))),
  ( 'add_index',
    Index('ix_meteringlabels_project_id', Column('project_id', String(length=255), table=<meteringlabels>))),
  ( 'add_index',
    Index('ix_ml2_gre_allocations_allocated', Column('allocated', Boolean(), table=<ml2_gre_allocations>, nullable=False, default=ColumnDefault(False), server_default=DefaultClause(<sqlalchemy.sql.elements.False_ object at 0x7f03dcbe6d30>, for_update=False)))),
  ( 'add_index',
    Index('ix_ml2_vxlan_allocations_allocated', Column('allocated', Boolean(), table=<ml2_vxlan_allocations>, nullable=False, default=ColumnDefault(False), server_default=DefaultClause(<sqlalchemy.sql.elements.False_ object at 0x7f03dcb86310>, for_update=False)))),
  ( 'add_index',
    Index('ix_networkdnsdomains_network_id', Column('network_id', String(length=36), table=<networkdnsdomains>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_networkrbacs_project_id', Column('project_id', String(length=255), table=<networkrbacs>))),
  ( 'add_index',
    Index('ix_networks_project_id', Column('project_id', String(length=255), table=<networks>))),
  ( 'add_index',
    Index('ix_ovn_hash_ring_group_name', Column('group_name', String(length=256), table=<ovn_hash_ring>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_hash_ring_node_uuid', Column('node_uuid', String(length=36), table=<ovn_hash_ring>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_revision_numbers_resource_type', Column('resource_type', String(length=36), table=<ovn_revision_numbers>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_revision_numbers_resource_uuid', Column('resource_uuid', String(length=36), table=<ovn_revision_numbers>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_portdataplanestatuses_port_id', Column('port_id', String(length=36), table=<portdataplanestatuses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_portdnses_port_id', Column('port_id', String(length=36), table=<portdnses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ports_project_id', Column('project_id', String(length=255), table=<ports>))),
  ( 'add_index',
    Index('ix_portuplinkstatuspropagation_port_id', Column('port_id', String(length=36), table=<portuplinkstatuspropagation>, primary_key=True, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('qos_policy_id', NullType(), table=<qos_dscp_marking_rules>))),
  ( 'add_constraint',
    UniqueConstraint(Column('fip_id', NullType(), table=<qos_fip_policy_bindings>))),
  ( 'add_index',
    Index('ix_qos_minimum_bandwidth_rules_qos_policy_id', Column('qos_policy_id', String(length=36), table=<qos_minimum_bandwidth_rules>, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('network_id', NullType(), table=<qos_network_policy_bindings>))),
  ( 'add_index',
    Index('ix_qos_policies_project_id', Column('project_id', String(length=255), table=<qos_policies>))),
  ( 'add_index',
    Index('ix_qos_policies_default_project_id', Column('project_id', String(length=255), table=<qos_policies_default>, primary_key=True, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('port_id', NullType(), table=<qos_port_policy_bindings>))),
  ( 'add_constraint',
    UniqueConstraint(Column('router_id', NullType(), table=<qos_router_gw_policy_bindings>))),
  ( 'add_index',
    Index('ix_quotas_project_id', Column('project_id', String(length=255), table=<quotas>))),
  ( 'add_index',
    Index('ix_quotausages_project_id', Column('project_id', String(length=255), table=<quotausages>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_quotausages_resource', Column('resource', String(length=255), table=<quotausages>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_routers_project_id', Column('project_id', String(length=255), table=<routers>))),
  ( 'add_index',
    Index('ix_securitygrouprules_project_id', Column('project_id', String(length=255), table=<securitygrouprules>))),
  ( 'add_index',
    Index('ix_securitygroups_project_id', Column('project_id', String(length=255), table=<securitygroups>))),
  ( 'add_index',
    Index('ix_segmenthostmappings_host', Column('host', String(length=255), table=<segmenthostmappings>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_segmenthostmappings_segment_id', Column('segment_id', String(length=36), table=<segmenthostmappings>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_subnet_dns_publish_fixed_ips_subnet_id', Column('subnet_id', String(length=36), table=<subnet_dns_publish_fixed_ips>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_subnetpools_project_id', Column('project_id', String(length=255), table=<subnetpools>))),
  ( 'add_index',
    Index('ix_subnets_project_id', Column('project_id', String(length=255), table=<subnets>))),
  ( 'add_index',
    Index('ix_subnets_subnetpool_id', Column('subnetpool_id', String(length=36), table=<subnets>))),
  ( 'add_index',
    Index('ix_trunks_project_id', Column('project_id', String(length=255), table=<trunks>)))]

[LIU Yulong (dragon889)]

++
https://c8bc63fd8c0b84666835-fef50fbf80bf7a8da07d35d3242576b8.ssl.cf5.rackcdn.com/773283/24/check/neutron-functional-with-uwsgi/fbd41bf/testr_results.html

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

The problem seems to be in the alembic version >= 1.6.3 [1]. When using alembic==1.6.2, test pass.

Still investigating.

Regards.

[1]https://review.opendev.org/c/openstack/requirements/+/792733/4/upper-constraints.txt#537

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Ok, I've tracked down the issue to https://github.com/sqlalchemy/alembic/commit/acc82bff397be712a7e4f5e2069e7bfbf93ff5a9.

I'm talking to the author (and project maintainer) to see how to address this.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

And I think the issue will be solved in https://github.com/sqlalchemy/alembic/issues/848.

For now, I'll disable those tests.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/792945

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/792945
Reason: We have combined this patch in https://review.opendev.org/c/openstack/neutron/+/791365 to fix the CI in one shot.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791365
Committed: https://opendev.org/openstack/neutron/commit/4de350a54a3e3e59371391a7a022447b49ae508e
Submitter: "Zuul (22348)"
Branch: master

commit 4de350a54a3e3e59371391a7a022447b49ae508e
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 14 08:31:32 2021 +0200

    Switch neutron-rally-task job to ML2/OVS and skip "test_models_sync"

    After switch of the default Neutron backend to the ML2/OVN
    in the Devstack we had 2 rally jobs running actually with the
    same Neutron backend.

    This patch configures explicitly ML2/OVS as a backend in the
    'neutron-rally-task' job so it is like it was before the change in
    the Devstack repo.

    In order to fix the CI, this patch also skips "test_models_sync" test
    cases. Once [1] is fixed and released, we'll be able to create SQL
    tables specifying the "_constraints_included" flag. When creating a
    table, this flag should be set to "False" in order to be able to set
    the "index" and "unique" parameters for each column in the table
    definition, as we do now.

    "test_update_router_admin_state" is excluded from
    "neutron-ovn-tempest-slow" until LP#1890445 is fixed.

    Set "test_restart_wsgi_on_sighup_multiple_workers" as unstable.

    CI job "neutron-ovn-tempest-slow" has been set to non-voting until
    LP#1930402 is fixed.

    Related-Bug: #1929518
    Related-Bug: #1890445
    Related-Bug: #1930402
    Related-Bug: #1930367

    [1]https://github.com/sqlalchemy/alembic/issues/848

    Change-Id: I38ed7931169434fecbbefa3762f51a522ef3c509

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795058

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795539

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795539

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/795058
Committed: https://opendev.org/openstack/neutron/commit/c915b93e7690f6d8accf95417284600df9aa5a90
Submitter: "Zuul (22348)"
Branch: master

commit c915b93e7690f6d8accf95417284600df9aa5a90
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 7 08:20:29 2021 +0000

    Bump alembic to 1.6.5

    This version fixes the problem reported in LP#1929518 as described
    in [1].

    [1]https://github.com/sqlalchemy/alembic/commit/acc82bff397be712a7e4f5e2069e7bfbf93ff5a9

    Change-Id: I894eb3524681a3abaca5eeca0aa47028e7f1ff44
    Closes-Bug: #1929518

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.