'Comparator' object has no attribute '_get_table' POST_FAILURE in ansible-collections-openstack-functional-devstack-ansible jobs

[Ananya Banerjee]

We see POST_FAILURE in ansible-collections-openstack-functional-devstack-ansible-2.11 and ansible-collections-openstack-functional-devstack-ansible-2.9 with error AttributeError: 'Comparator' object has no attribute '_get_table'

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ab7/periodic/opendev.org/openstack/ansible-collections-openstack/master/ansible-collections-openstack-functional-devstack-ansible-2.11/ab76bd6/job-output.txt

https://0c6eceddee3cfc5105b9-48ea8802f13e15aef44645049d82e532.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/ansible-collections-openstack/master/ansible-collections-openstack-functional-devstack-ansible-2.9/1e25abb/job-output.txt

2021-06-14 06:53:31.361122 | controller | DEBUG migrate.versioning.script.base [None designate-manage None None] Loading script /opt/stack/designate/designate/storage/impl_sqlalchemy/migrate_repo/versions/073_placeholder.py... {{(pid=118565) __init__ /usr/local/lib/python3.8/dist-packages/migrate/versioning/script/base.py:27}}
2021-06-14 06:53:31.361411 | controller | DEBUG migrate.versioning.script.base [None designate-manage None None] Script /opt/stack/designate/designate/storage/impl_sqlalchemy/migrate_repo/versions/073_placeholder.py loaded successfully {{(pid=118565) __init__ /usr/local/lib/python3.8/dist-packages/migrate/versioning/script/base.py:30}}
2021-06-14 06:53:31.361680 | controller | DEBUG migrate.versioning.repository [None designate-manage None None] Repository /opt/stack/designate/designate/storage/impl_sqlalchemy/migrate_repo loaded successfully {{(pid=118565) __init__ /usr/local/lib/python3.8/dist-packages/migrate/versioning/repository.py:82}}
2021-06-14 06:53:31.361955 | controller | DEBUG migrate.versioning.repository [None designate-manage None None] Config: {'db_settings': {'repository_id': 'Designate', 'version_table': 'migrate_version', 'required_dbs': '[]', 'use_timestamp_numbering': 'False'}} {{(pid=118565) __init__ /usr/local/lib/python3.8/dist-packages/migrate/versioning/repository.py:83}}
2021-06-14 06:53:31.373284 | controller | INFO migrate.versioning.api [None designate-manage None None] 69 -> 70... 
2021-06-14 06:53:32.437469 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.437674 | controller | INFO migrate.versioning.api [None designate-manage None None] 70 -> 71... 
2021-06-14 06:53:32.448435 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.448739 | controller | INFO migrate.versioning.api [None designate-manage None None] 71 -> 72... 
2021-06-14 06:53:32.459156 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.459477 | controller | INFO migrate.versioning.api [None designate-manage None None] 72 -> 73... 
2021-06-14 06:53:32.469896 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.470208 | controller | INFO migrate.versioning.api [None designate-manage None None] 73 -> 74... 
2021-06-14 06:53:32.480417 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.480589 | controller | INFO migrate.versioning.api [None designate-manage None None] 74 -> 75... 
2021-06-14 06:53:32.492012 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.492356 | controller | INFO migrate.versioning.api [None designate-manage None None] 75 -> 76... 
2021-06-14 06:53:32.503886 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.504146 | controller | INFO migrate.versioning.api [None designate-manage None None] 76 -> 77... 
2021-06-14 06:53:32.515286 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.515593 | controller | INFO migrate.versioning.api [None designate-manage None None] 77 -> 78... 
2021-06-14 06:53:32.526482 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.526792 | controller | INFO migrate.versioning.api [None designate-manage None None] 78 -> 79... 
2021-06-14 06:53:32.536860 | controller | INFO migrate.versioning.api [None designate-manage None None] done
2021-06-14 06:53:32.537149 | controller | INFO migrate.versioning.api [None designate-manage None None] 79 -> 80... 
2021-06-14 06:53:32.691710 | controller | CRITICAL designate [None designate-manage None None] Unhandled error: oslo_db.exception.DBMigrationError: Neither 'Column' object nor 'Comparator' object has an attribute '_get_table'
2021-06-14 06:53:32.691947 | controller | ERROR designate Traceback (most recent call last):
2021-06-14 06:53:32.692076 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 832, in __getattr__
2021-06-14 06:53:32.692191 | controller | ERROR designate  return getattr(self.comparator, key)
2021-06-14 06:53:32.692330 | controller | ERROR designate AttributeError: 'Comparator' object has no attribute '_get_table'
2021-06-14 06:53:32.692441 | controller | ERROR designate 
2021-06-14 06:53:32.692570 | controller | ERROR designate The above exception was the direct cause of the following exception:
2021-06-14 06:53:32.692707 | controller | ERROR designate 
2021-06-14 06:53:32.692849 | controller | ERROR designate Traceback (most recent call last):
2021-06-14 06:53:32.692962 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/migration.py", line 87, in db_sync
2021-06-14 06:53:32.693070 | controller | ERROR designate  migration = versioning_api.upgrade(engine, repository, version)
2021-06-14 06:53:32.693175 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/migrate/versioning/api.py", line 186, in upgrade
2021-06-14 06:53:32.693281 | controller | ERROR designate  return _migrate(url, repository, version, upgrade=True, err=err, **opts)
2021-06-14 06:53:32.693387 | controller | ERROR designate  File "<decorator-gen-15>", line 2, in _migrate
2021-06-14 06:53:32.693501 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/migrate/versioning/util/__init__.py", line 167, in with_engine
2021-06-14 06:53:32.693624 | controller | ERROR designate  return f(*a, **kw)
2021-06-14 06:53:32.693731 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/migrate/versioning/api.py", line 366, in _migrate
2021-06-14 06:53:32.693880 | controller | ERROR designate  schema.runchange(ver, change, changeset.step)
2021-06-14 06:53:32.693997 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/migrate/versioning/schema.py", line 93, in runchange
2021-06-14 06:53:32.694103 | controller | ERROR designate  change.run(self.engine, step)
2021-06-14 06:53:32.694209 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/migrate/versioning/script/py.py", line 154, in run
2021-06-14 06:53:32.694312 | controller | ERROR designate  script_func(engine)
2021-06-14 06:53:32.694553 | controller | ERROR designate  File "/opt/stack/designate/designate/storage/impl_sqlalchemy/migrate_repo/versions/080_domain_to_zone_rename.py", line 108, in upgrade
2021-06-14 06:53:32.694773 | controller | ERROR designate  drop_foreign_key(fk)
2021-06-14 06:53:32.695061 | controller | ERROR designate  File "/opt/stack/designate/designate/storage/impl_sqlalchemy/migrate_repo/versions/080_domain_to_zone_rename.py", line 52, in drop_foreign_key
2021-06-14 06:53:32.695192 | controller | ERROR designate  table = fk_def[0]._get_table()
2021-06-14 06:53:32.695306 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 834, in __getattr__
2021-06-14 06:53:32.695415 | controller | ERROR designate  util.raise_(
2021-06-14 06:53:32.695562 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 211, in raise_
2021-06-14 06:53:32.695680 | controller | ERROR designate  raise exception
2021-06-14 06:53:32.695788 | controller | ERROR designate AttributeError: Neither 'Column' object nor 'Comparator' object has an attribute '_get_table'
2021-06-14 06:53:32.695892 | controller | ERROR designate 
2021-06-14 06:53:32.695997 | controller | ERROR designate During handling of the above exception, another exception occurred:
2021-06-14 06:53:32.696100 | controller | ERROR designate 
2021-06-14 06:53:32.696203 | controller | ERROR designate Traceback (most recent call last):
2021-06-14 06:53:32.696306 | controller | ERROR designate  File "/usr/local/bin/designate-manage", line 10, in <module>
2021-06-14 06:53:32.696410 | controller | ERROR designate  sys.exit(main())
2021-06-14 06:53:32.696514 | controller | ERROR designate  File "/opt/stack/designate/designate/cmd/manage.py", line 123, in main
2021-06-14 06:53:32.696616 | controller | ERROR designate  fn(*fn_args)
2021-06-14 06:53:32.696759 | controller | ERROR designate  File "/opt/stack/designate/designate/manage/database.py", line 49, in sync
2021-06-14 06:53:32.696873 | controller | ERROR designate  get_manager().upgrade(None)
2021-06-14 06:53:32.697100 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/migration_cli/manager.py", line 72, in upgrade
2021-06-14 06:53:32.697353 | controller | ERROR designate  results.append(plugin.upgrade(None))
2021-06-14 06:53:32.697499 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/migration_cli/ext_migrate.py", line 45, in upgrade
2021-06-14 06:53:32.697611 | controller | ERROR designate  return migration.db_sync(
2021-06-14 06:53:32.697721 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/debtcollector/removals.py", line 242, in wrapper
2021-06-14 06:53:32.697826 | controller | ERROR designate  return f(*args, **kwargs)
2021-06-14 06:53:32.697935 | controller | ERROR designate  File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/migration.py", line 89, in db_sync
2021-06-14 06:53:32.698039 | controller | ERROR designate  raise exception.DBMigrationError(ex)
2021-06-14 06:53:32.698163 | controller | ERROR designate oslo_db.exception.DBMigrationError: Neither 'Column' object nor 'Comparator' object has an attribute '_get_table'
2021-06-14 06:53:32.698268 | controller | ERROR designate 
2021-06-14 06:53:32.943171 | controller | Error on exit
2021-06-14 06:53:34.154943 | controller | ebtables v1.8.4 (nf_tables): table `broute' is incompatible, use 'nft' tool.

[Ananya Banerjee (frenzyfriday)]

To track the failing jobs

https://zuul.openstack.org/builds?job_name=ansible-collections-openstack-functional-devstack-ansible-2.11&job_name=ansible-collections-openstack-functional-devstack-ansible-2.9&job_name=ansible-collections-openstack-functional-devstack&job_name=ansible-collections-openstack-functional-devstack-ansible-devel&job_name=ansible-collections-openstack-functional-devstack-octavia&job_name=ansible-collections-openstack-functional-devstack-releases&job_name=ansible-collections-openstack-functional-devstack-queens-ansible-2.11

[Ananya Banerjee (frenzyfriday)]

Closing as we are no longer hitting this bug.