Master standalone deploy failed with Table 'ovn_revision_numbers' already exists while performing neutron sync

[chandan kumar]

Master periodic standalone deploy got failed with following error while performing neutron db migration
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/668be38/logs/undercloud/var/log/extra/podman/containers/neutron_db_sync/stdout.log.txt.gz

File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1066, in read
    first_packet = self.connection._read_packet()
  File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 683, in _read_packet
    packet.check_error()
  File "/usr/lib/python2.7/site-packages/pymysql/protocol.py", line 220, in check_error
    err.raise_mysql_exception(self._data)
  File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 109, in raise_mysql_exception
    raise errorclass(errno, errval)
oslo_db.exception.DBError: (pymysql.err.InternalError) (1050, u"Table 'ovn_revision_numbers' already exists") [SQL: u'\nCREATE TABLE ovn_revision_numbers (\n\tstandard_attr_id BIGINT, \n\tresource_uuid VARCHAR(36) NOT NULL, \n\tresource_type VARCHAR(36) NOT NULL, \n\trevision_number BIGINT NOT NULL, \n\tcreated_at DATETIME NOT NULL, \n\tupdated_at TIMESTAMP NULL, \n\tPRIMARY KEY (resource_uuid), \n\tFOREIGN KEY(standard_attr_id) REFERENCES standardattributes (id) ON DELETE SET NULL\n)ENGINE=InnoDB\n\n'] (Background on this error at: http://sqlalche.me/e/2j85)

https://review.opendev.org/#/c/695969/ ->[OVN] Add DB tables for OVN backend in neutron might cause the issue.

Few more logs from other jobs:

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-master/d9a7a79/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario003-standalone-master/a69d44f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/6332454/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

[chandan kumar (chkumar246)]

It also impacted multinode jobs
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/47f68e0/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master/ef04624/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/1eaa08c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/4b41039/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

[Marios Andreou (marios-b)]

confirmed it could well be that https://review.opendev.org/#/c/695969/ ... looking at master build report and neutron hash it includes that commit

        * https://trunk.rdoproject.org/centos7-master-head/report.html
            * 2019-12-03 08:22:46 2019-12-02 18:59:05 openstack-neutron
991126eb6ea852f578b6d1d4dcc88de1af5d3029

        * https://github.com/openstack/neutron/commits/master

[Marios Andreou (marios-b)]

"Skip DB table creation and model definition" https://review.opendev.org/#/c/697039/ Change-Id: Id625b0abab09fda022be23954ffca453499566bd openstack/networking-ovn

is the fix for this

[yatin (yatinkarel)]

Also hit packstack https://centos.logs.rdoproject.org/weirdo-generic-packstack-scenario-py3/1903/weirdo-project/logs/latest/openstack-setup.log.txt.gz
https://centos.logs.rdoproject.org/weirdo-generic-packstack-scenario001/7012/weirdo-project/logs/latest/openstack-setup.log.txt.gz

[Marios Andreou (marios-b)]

patch @ https://review.opendev.org/#/c/697039/ merged now but didn't get a green run yet so keeping it open until we do e.g. there https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-master&job_name=periodic-tripleo-ci-centos-7-scenario001-standalone-master

[Marios Andreou (marios-b)]

just did some more digging because the periodics from this morning failed on that in [1][2][3]

The problem is we didn't get a delorean consistent build yet. The consistent [4] and tripleo-ci-testing [5] (this one is used in periodics) still use

        * python-networking-ovn-7.0.0-0.20191128183743.7c8bbd2.el7.src.rpm

We need a new consistent build so that the current [6] build can end up in the periodics. current has the right version

        * python-networking-ovn-7.0.0-0.20191205000005.3c0225c.el7.src.rpm

in particular you can confirm by checking the .3c0225c commit @ [7] which is the one we're interested in for the fix

[1] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario004-standalone-master/d172f45/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[2] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-scenario002-standalone-master/b50ffe8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[3] http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-master/4e2d5c8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
[4] https://trunk.rdoproject.org/centos7/consistent/
[5] https://trunk.rdoproject.org/centos7/tripleo-ci-testing/
[6] https://trunk.rdoproject.org/centos7/current/
[7] https://opendev.org/openstack/networking-ovn/commits/branch/master

[Marios Andreou (marios-b)]

looks good now https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-standalone-master&job_name=periodic-tripleo-ci-centos-7-scenario001-standalone-master

and consistent build/tripleo-ci-testing

        * https://trunk.rdoproject.org/centos7/consistent/
            * python-networking-ovn-7.0.0-0.20191205000005.3c0225c.el7.src.rpm 2019-12-05 00:05 1.2M

        * https://trunk.rdoproject.org/centos7/tripleo-ci-testing/
            * python-networking-ovn-7.0.0-0.20191205000005.3c0225c.el7.src.rpm 2019-12-05 00:05 1.2M

moving fix-released