dbsync fails in newton

[Gabriele Cerami]

neutron dbsync fails in newton while trying to install neutron with tripleo.

the short error is

oslo_db.exception.DBError: (pymysql.err.InternalError) (1060, u"Duplicate column name 'tenant_id'") [SQL: u'ALTER TABLE bsn_routerrules ADD COLUMN tenant_id VARCHAR(255)']

complete traceback with errors can be found at

http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-ha-newton/8ae2144/logs/postci.txt.gz#_2017-01-23_07_36_43_000

The error appears to be caused by

https://review.openstack.org/gitweb?p=openstack/networking-bigswitch.git;a=commit;h=c5ec4ff80e8fc7b8872134c17a123cf3500aa341

line 36 at

https://review.openstack.org/gitweb?p=openstack/networking-bigswitch.git;a=blob;f=bsnstacklib/plugins/bigswitch/db/migration/alembic_migrations/versions/34f888285695_support_multiple_logical_routers.py;h=e869315f3198a06b551ce633c12b17a9975a8552;hb=c5ec4ff80e8fc7b8872134c17a123cf3500aa341#l36

doesn't seem to check if the column "tenant_id" already exists.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424134

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/424152

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/424172

[Gabriele Cerami (gcerami)]

From what I read in alembic documentation, manually adding a column is not the recommended way to handle a migration: the preferred way is to define the new scheme entirely and then feed alembic the diff with the old scheme.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424331

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424376

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/424378

[OpenStack Infra (hudson-openstack)]

Change abandoned by Ben Nemec (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/424378
Reason: Emilien just beat me to it: https://review.openstack.org/#/c/424376

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424376
Committed: https://git.openstack.org/cgit/openstack/networking-bigswitch/commit/?id=83986a11d6d4df8c3ba75e758fb6692c9c215c64
Submitter: Jenkins
Branch: stable/newton

commit 83986a11d6d4df8c3ba75e758fb6692c9c215c64
Author: Emilien Macchi <email address hidden>
Date: Mon Jan 23 23:47:12 2017 +0000

    Revert "OSP-6 support MLR in bsnstacklib"

    This reverts commit c5ec4ff80e8fc7b8872134c17a123cf3500aa341.

    This backport broke Newton deployments in TripleO.
    Bigswitch team shouldn't backport features into stable branches, it's breaking more than you think.

    In our case, you can't deploy TripleO anymore, see the bug report:
    https://bugs.launchpad.net/networking-bigswitch/+bug/1658645

    Change-Id: I3d48e11021066cd8d71d8510585db85c03b09e1e
    Partial-Bug: #1658645

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424172
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=6dcfef81fbeb37f42b72181bf547a2ce5859a799
Submitter: Jenkins
Branch: master

commit 6dcfef81fbeb37f42b72181bf547a2ce5859a799
Author: Gabriele Cerami <email address hidden>
Date: Mon Jan 23 15:40:52 2017 +0100

    Remove vendor plugins from image build

    mirroring what is done for master and newton in tripleo-common [1]
    we are removing vendor plugins from default image build process, because
    they are proven to be unreliable, difficult to gate and slow to fix.

    [1] https://review.openstack.org/409809

    Change-Id: I52371311d02d987ec198380ec268addd27f36684
    Closes-Bug: 1658645

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424152
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=d9efb1f3311e3b8655cec335193c0f15e63c1ab9
Submitter: Jenkins
Branch: stable/newton

commit d9efb1f3311e3b8655cec335193c0f15e63c1ab9
Author: John Trowbridge <email address hidden>
Date: Mon Dec 12 09:19:14 2016 -0500

    Remove remaining vendor plugins from default image YAML

    Trunk is again broken by a vendor plugin not keeping up with
    deprecations in neutron. This time it is the bigswitch plugin
    that is using code from neutron that was moved to neutron-lib.

    There is not really any reason to have these vendor plugins
    installed in the default YAML. If the vendor wants to build images
    for TripleO in their own CI, it is simply a matter of passing an
    extra yaml with the packages key.

    Change-Id: Ib7c10f2dbd692494bd1990c3a2988308096743e7
    Closes-Bug: 1649284
    (cherry picked from commit ec78e9d84d13de60b54368dd751a338d8c9af73b)
    Related-Bug: 1658645

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424331
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=fa941f1542c55e09d3d4fa71d7698282c1437d64
Submitter: Jenkins
Branch: stable/newton

commit fa941f1542c55e09d3d4fa71d7698282c1437d64
Author: Gabriele Cerami <email address hidden>
Date: Mon Jan 23 15:40:52 2017 +0100

    Remove vendor plugins from image build

    mirroring what is done for master and newton in tripleo-common [1]
    we are removing vendor plugins from default image build process, because
    they are proven to be unreliable, difficult to gate and slow to fix.

    [1] https://review.openstack.org/409809

    (cherry picked from commit I52371311d02d987ec198380ec268addd27f36684)
    Change-Id: I52371311d02d987ec198380ec268addd27f36684
    Closes-Bug: 1658645

[OpenStack Infra (hudson-openstack)]

Change abandoned by Gabriele Cerami (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/424134
Reason: obsoleted

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 6.0.0 release.

[Aditya Vaja (wolverine-av)]

Hi,

Apologies for the delayed response. DB upgrade scripts are idempotent, so this shouldn't have happened ideally. I'm not sure how the column exists, since this is the change where it was added. And I did test it by upgrading multiple times.

I see that vendor plugins are now removed from tripleo-common, does that mean this can be safely merged and any breakouts can be fixed on our end without messing upstream?

[Alfredo Moralejo (amoralej)]

We are hitting this again after https://review.openstack.org/#/c/433232/ in RDO-CI:

https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-minimal-328/overcloud-controller-0/var/log/messages.gz

Somehow we are still getting bigswitch installed in overcloud images [1] even after https://review.openstack.org/424331

[1] https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-promote-newton-delorean-minimal-328/overcloud-controller-0/var/log/extra/rpm-list.txt.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/434801

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alfredo Moralejo (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/434801
Reason: I'm not going to ask revert of all subsequents changes and i guess i'll mess it up with manual fixing all conflicts so i'll leave project maintainer to fix this.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 5.4.2 release.