nailgun_fixtures not uploaded to db if db was restored from 7.0. it breaks upgrade

[Sergey Abramov]

1) restore 7.0 data from pg_dump
2) nailgun_syncdb
3) nailgun_fixtures

2015-12-22 15:16:15.782 INFO [7fdaedcc1740] (fixman) Fixture model 'release' with pk='1' already uploaded. Skipping
2015-12-22 15:16:15.788 INFO [7fdaedcc1740] (fixman) Fixture model 'release' with pk='2' already uploaded. Skipping
2015-12-22 15:16:15.788 INFO [7fdaedcc1740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixtures/openstack.yaml
2015-12-22 15:16:15.791 INFO [7fdaedcc1740] (fixman) Fixture model 'notification' with pk='1' already uploaded. Skipping
2015-12-22 15:16:15.792 INFO [7fdaedcc1740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixtures/start_notification.json
2015-12-22 15:16:15.800 INFO [7fdaedcc1740] (fixman) Fixture model 'network_group' with pk='1' already uploaded. Skipping
2015-12-22 15:16:15.801 INFO [7fdaedcc1740] (fixman) Fixture model 'i_p_addr_range' with pk='1' already uploaded. Skipping
2015-12-22 15:16:15.803 INFO [7fdaedcc1740] (fixman) Fixture model 'i_p_addr' with pk='1' already uploaded. Skipping

[root@fuel ~]# fuel release
id | name | state | operating_system | version
---|----------------------|-------------|------------------|-------------
2 | Kilo on Ubuntu 14.04 | manageonly | Ubuntu | 2015.1.0-7.0
1 | Kilo on CentOS 6.5 | unavailable | CentOS | 2015.1.0-7.0

[Sergey Abramov (sabramov)]

problem is in check

file nailgun/db/sqlalchemy/fixman.py

        # Check if it's already uploaded
        obj_from_db = db().query(obj['model']).get(pk)
        if obj_from_db:
            logger.info("Fixture model '%s' with pk='%s' already"
                        " uploaded. Skipping", model_name, pk)
            continue

Existence of pk is not good way to check fixtures upload

[Sergey Abramov (sabramov)]

I removed this check. It seems to me, that this is resolve my problem.

[root@fuel ~]# nailgun_fixtures
2015-12-22 16:23:29.378 DEBUG [7fb0b65dc740] (settings) Looking for settings.yaml package config using old style __file__
2015-12-22 16:23:29.379 DEBUG [7fb0b65dc740] (settings) Trying to read config file /usr/lib/python2.7/site-packages/nailgun/settings.yaml
2015-12-22 16:23:29.573 DEBUG [7fb0b65dc740] (settings) Trying to read config file /etc/nailgun/settings.yaml
/usr/lib64/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value 'This option will install ...'. (this warning may be suppressed after 10 occurrences)
  (util.ellipses_string(value),))
/usr/lib64/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value 'Kilo on CentOS 6.5'. (this warning may be suppressed after 10 occurrences)
  (util.ellipses_string(value),))
/usr/lib64/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value 'Kilo on Ubuntu 14.04'. (this warning may be suppressed after 10 occurrences)
  (util.ellipses_string(value),))
2015-12-22 16:23:30.552 INFO [7fb0b65dc740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixtures/openstack.yaml
2015-12-22 16:23:30.557 INFO [7fb0b65dc740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixtures/start_notification.json
2015-12-22 16:23:30.575 INFO [7fb0b65dc740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixtures/admin_network.json
[root@fuel ~]# vi /usr/lib/python2.7/site-packages/nailgun/db/sqlalchemy/fixman.py
[root@fuel ~]# fuel release
id | name | state | operating_system | version
---|----------------------|-------------|------------------|-------------
4 | Kilo on Ubuntu 14.04 | available | Ubuntu | 2015.1.0-8.0
3 | Kilo on CentOS 6.5 | unavailable | CentOS | 2015.1.0-8.0
2 | Kilo on Ubuntu 14.04 | manageonly | Ubuntu | 2015.1.0-7.0
1 | Kilo on CentOS 6.5 | unavailable | CentOS | 2015.1.0-7.0

[Alexander Kislitsky (akislitsky)]

If it breaks upgrade it is High priority bug.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260955

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/260955
Reason: Would be fixed by removing hardcoded pk values from fixtures.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/260564
Reason: Decided to fix an issues with minimal changes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262018
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d3270d427b60c8d352bb2d78711e569ddc5f1903
Submitter: Jenkins
Branch: master

commit d3270d427b60c8d352bb2d78711e569ddc5f1903
Author: Alexander Kislitsky <email address hidden>
Date: Mon Dec 28 18:21:33 2015 +0300

    Uploading fixtures only for clear DB introduced

    We should upload fixtures only once at filling the empty DB.
    Applying fixtures on the production DB can mess the data.

    Change-Id: I88a90230e7898061f4ef0ed094f53a3fba4702fa
    Partial-Bug: #1526917
    Closes-Bug: #1528620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/268091

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268091
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=be76f73fd2444b1b970ecb196aea299b51a3e82c
Submitter: Jenkins
Branch: stable/8.0

commit be76f73fd2444b1b970ecb196aea299b51a3e82c
Author: Alexander Kislitsky <email address hidden>
Date: Mon Dec 28 18:21:33 2015 +0300

    Uploading fixtures only for clear DB introduced

    We should upload fixtures only once at filling the empty DB.
    Applying fixtures on the production DB can mess the data.

    Change-Id: I88a90230e7898061f4ef0ed094f53a3fba4702fa
    Partial-Bug: #1526917
    Closes-Bug: #1528620
    (cherry picked from commit d3270d427b60c8d352bb2d78711e569ddc5f1903)

[Nastya Urlapova (aurlapova)]

[root@nailgun nailgun]# manage.py loaddefault
2016-02-16 14:38:23.502 DEBUG [7f1637e96740] (settings) Looking for settings.yaml package config using old style __file__
2016-02-16 14:38:23.502 DEBUG [7f1637e96740] (settings) Trying to read config file /usr/lib/python2.7/site-packages/nailgun/settings.yaml
2016-02-16 14:38:23.814 DEBUG [7f1637e96740] (settings) Trying to read config file /etc/nailgun/settings.yaml
2016-02-16 14:38:24.440 INFO [7f1637e96740] (manage) Uploading fixture...
2016-02-16 14:38:24.644 INFO [7f1637e96740] (fixman) Fixtures are already uploaded. Skipping
2016-02-16 14:38:24.644 INFO [7f1637e96740] (manage) Applying fake deployment tasks to all releases...
2016-02-16 14:38:24.745 INFO [7f1637e96740] (manage) Done

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "529"
  build_id: "529"
  fuel-nailgun_sha: "baec8643ca624e52b37873f2dbd511c135d236d9"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "e2d79330d5d708796330fac67722c21f85569b87"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "e071216cb214e34b4d861478033425ee6a54a3be"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"