Functional db migration tests broken

Bug #1929518 reported by Slawek Kaplonski
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
neutron
Critical
Rodolfo Alonso

Bug Description

It seems that it's failing all the time now. Example of failure https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ba6/790999/7/check/neutron-functional-with-uwsgi/ba6f15c/testr_results.html

Stacktrace:

ft1.4: neutron.tests.functional.db.test_migrations.TestModelsMigrationsMysql.test_models_synctesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 125, in inner
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/db/test_migrations.py", line 385, in test_models_sync
    super(TestModelsMigrationsMysql, self).test_models_sync()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/oslo_db/sqlalchemy/test_migrations.py", line 597, in test_models_sync
    self.fail(
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.8/site-packages/unittest2/case.py", line 690, in fail
    raise self.failureException(msg)
AssertionError: Models and migration scripts aren't in sync:
[ ( 'add_index',
    Index('ix_address_groups_project_id', Column('project_id', String(length=255), table=<address_groups>))),
  ( 'add_index',
    Index('ix_address_scopes_project_id', Column('project_id', String(length=255), table=<address_scopes>))),
  ( 'add_index',
    Index('ix_addressgrouprbacs_project_id', Column('project_id', String(length=255), table=<addressgrouprbacs>))),
  ( 'add_constraint',
    UniqueConstraint(Column('mac_address', NullType(), table=<dvr_host_macs>))),
  ( 'add_index',
    Index('ix_floatingipdnses_floatingip_id', Column('floatingip_id', String(length=36), table=<floatingipdnses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_floatingips_project_id', Column('project_id', String(length=255), table=<floatingips>))),
  ( 'add_index',
    Index('ix_logs_project_id', Column('project_id', String(length=255), table=<logs>))),
  ( 'add_index',
    Index('ix_logs_resource_id', Column('resource_id', String(length=36), table=<logs>))),
  ( 'add_index',
    Index('ix_logs_target_id', Column('target_id', String(length=36), table=<logs>))),
  ( 'add_index',
    Index('ix_meteringlabels_project_id', Column('project_id', String(length=255), table=<meteringlabels>))),
  ( 'add_index',
    Index('ix_ml2_gre_allocations_allocated', Column('allocated', Boolean(), table=<ml2_gre_allocations>, nullable=False, default=ColumnDefault(False), server_default=DefaultClause(<sqlalchemy.sql.elements.False_ object at 0x7f03dcbe6d30>, for_update=False)))),
  ( 'add_index',
    Index('ix_ml2_vxlan_allocations_allocated', Column('allocated', Boolean(), table=<ml2_vxlan_allocations>, nullable=False, default=ColumnDefault(False), server_default=DefaultClause(<sqlalchemy.sql.elements.False_ object at 0x7f03dcb86310>, for_update=False)))),
  ( 'add_index',
    Index('ix_networkdnsdomains_network_id', Column('network_id', String(length=36), table=<networkdnsdomains>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_networkrbacs_project_id', Column('project_id', String(length=255), table=<networkrbacs>))),
  ( 'add_index',
    Index('ix_networks_project_id', Column('project_id', String(length=255), table=<networks>))),
  ( 'add_index',
    Index('ix_ovn_hash_ring_group_name', Column('group_name', String(length=256), table=<ovn_hash_ring>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_hash_ring_node_uuid', Column('node_uuid', String(length=36), table=<ovn_hash_ring>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_revision_numbers_resource_type', Column('resource_type', String(length=36), table=<ovn_revision_numbers>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ovn_revision_numbers_resource_uuid', Column('resource_uuid', String(length=36), table=<ovn_revision_numbers>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_portdataplanestatuses_port_id', Column('port_id', String(length=36), table=<portdataplanestatuses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_portdnses_port_id', Column('port_id', String(length=36), table=<portdnses>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_ports_project_id', Column('project_id', String(length=255), table=<ports>))),
  ( 'add_index',
    Index('ix_portuplinkstatuspropagation_port_id', Column('port_id', String(length=36), table=<portuplinkstatuspropagation>, primary_key=True, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('qos_policy_id', NullType(), table=<qos_dscp_marking_rules>))),
  ( 'add_constraint',
    UniqueConstraint(Column('fip_id', NullType(), table=<qos_fip_policy_bindings>))),
  ( 'add_index',
    Index('ix_qos_minimum_bandwidth_rules_qos_policy_id', Column('qos_policy_id', String(length=36), table=<qos_minimum_bandwidth_rules>, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('network_id', NullType(), table=<qos_network_policy_bindings>))),
  ( 'add_index',
    Index('ix_qos_policies_project_id', Column('project_id', String(length=255), table=<qos_policies>))),
  ( 'add_index',
    Index('ix_qos_policies_default_project_id', Column('project_id', String(length=255), table=<qos_policies_default>, primary_key=True, nullable=False))),
  ( 'add_constraint',
    UniqueConstraint(Column('port_id', NullType(), table=<qos_port_policy_bindings>))),
  ( 'add_constraint',
    UniqueConstraint(Column('router_id', NullType(), table=<qos_router_gw_policy_bindings>))),
  ( 'add_index',
    Index('ix_quotas_project_id', Column('project_id', String(length=255), table=<quotas>))),
  ( 'add_index',
    Index('ix_quotausages_project_id', Column('project_id', String(length=255), table=<quotausages>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_quotausages_resource', Column('resource', String(length=255), table=<quotausages>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_routers_project_id', Column('project_id', String(length=255), table=<routers>))),
  ( 'add_index',
    Index('ix_securitygrouprules_project_id', Column('project_id', String(length=255), table=<securitygrouprules>))),
  ( 'add_index',
    Index('ix_securitygroups_project_id', Column('project_id', String(length=255), table=<securitygroups>))),
  ( 'add_index',
    Index('ix_segmenthostmappings_host', Column('host', String(length=255), table=<segmenthostmappings>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_segmenthostmappings_segment_id', Column('segment_id', String(length=36), table=<segmenthostmappings>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_subnet_dns_publish_fixed_ips_subnet_id', Column('subnet_id', String(length=36), table=<subnet_dns_publish_fixed_ips>, primary_key=True, nullable=False))),
  ( 'add_index',
    Index('ix_subnetpools_project_id', Column('project_id', String(length=255), table=<subnetpools>))),
  ( 'add_index',
    Index('ix_subnets_project_id', Column('project_id', String(length=255), table=<subnets>))),
  ( 'add_index',
    Index('ix_subnets_subnetpool_id', Column('subnetpool_id', String(length=36), table=<subnets>))),
  ( 'add_index',
    Index('ix_trunks_project_id', Column('project_id', String(length=255), table=<trunks>)))]

Revision history for this message
LIU Yulong (dragon889) wrote :
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

The problem seems to be in the alembic version >= 1.6.3 [1]. When using alembic==1.6.2, test pass.

Still investigating.

Regards.

[1]https://review.opendev.org/c/openstack/requirements/+/792733/4/upper-constraints.txt#537

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Ok, I've tracked down the issue to https://github.com/sqlalchemy/alembic/commit/acc82bff397be712a7e4f5e2069e7bfbf93ff5a9.

I'm talking to the author (and project maintainer) to see how to address this.

Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

And I think the issue will be solved in https://github.com/sqlalchemy/alembic/issues/848.

For now, I'll disable those tests.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/792945

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/792945
Reason: We have combined this patch in https://review.opendev.org/c/openstack/neutron/+/791365 to fix the CI in one shot.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791365
Committed: https://opendev.org/openstack/neutron/commit/4de350a54a3e3e59371391a7a022447b49ae508e
Submitter: "Zuul (22348)"
Branch: master

commit 4de350a54a3e3e59371391a7a022447b49ae508e
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 14 08:31:32 2021 +0200

    Switch neutron-rally-task job to ML2/OVS and skip "test_models_sync"

    After switch of the default Neutron backend to the ML2/OVN
    in the Devstack we had 2 rally jobs running actually with the
    same Neutron backend.

    This patch configures explicitly ML2/OVS as a backend in the
    'neutron-rally-task' job so it is like it was before the change in
    the Devstack repo.

    In order to fix the CI, this patch also skips "test_models_sync" test
    cases. Once [1] is fixed and released, we'll be able to create SQL
    tables specifying the "_constraints_included" flag. When creating a
    table, this flag should be set to "False" in order to be able to set
    the "index" and "unique" parameters for each column in the table
    definition, as we do now.

    "test_update_router_admin_state" is excluded from
    "neutron-ovn-tempest-slow" until LP#1890445 is fixed.

    Set "test_restart_wsgi_on_sighup_multiple_workers" as unstable.

    CI job "neutron-ovn-tempest-slow" has been set to non-voting until
    LP#1930402 is fixed.

    Related-Bug: #1929518
    Related-Bug: #1890445
    Related-Bug: #1930402
    Related-Bug: #1930367

    [1]https://github.com/sqlalchemy/alembic/issues/848

    Change-Id: I38ed7931169434fecbbefa3762f51a522ef3c509

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795058

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795539

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795539

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/795058
Committed: https://opendev.org/openstack/neutron/commit/c915b93e7690f6d8accf95417284600df9aa5a90
Submitter: "Zuul (22348)"
Branch: master

commit c915b93e7690f6d8accf95417284600df9aa5a90
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 7 08:20:29 2021 +0000

    Bump alembic to 1.6.5

    This version fixes the problem reported in LP#1929518 as described
    in [1].

    [1]https://github.com/sqlalchemy/alembic/commit/acc82bff397be712a7e4f5e2069e7bfbf93ff5a9

    Change-Id: I894eb3524681a3abaca5eeca0aa47028e7f1ff44
    Closes-Bug: #1929518

Changed in neutron:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 19.0.0.0rc1

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.