Nailgun fixtures are uploaded on each container start

[Artem Panchenko]

Fuel version info (8.0 build #297): http://paste.openstack.org/show/482061/

Currently puppet tries to upload Nailgun fixtures each time its container is started. This leads to errors and 4+ minutes delay between container start and actual Nailgun start:

2015-12-16 16:38:31 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns (debug): Exec try 1/50
2015-12-16 16:38:31 +0000 Exec[nailgun_syncdb](provider=posix) (debug): Executing '/usr/bin/nailgun_syncdb'
2015-12-16 16:38:31 +0000 Puppet (debug): Executing '/usr/bin/nailgun_syncdb'
2015-12-16 16:38:31 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb]/returns (notice): executed successfully
2015-12-16 16:38:31 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_syncdb] (debug): The container Class[Nailgun::Venv] will propagate my refresh event
2015-12-16 16:38:31 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Exec try 1/50
2015-12-16 16:38:31 +0000 Exec[nailgun_upload_fixtures](provider=posix) (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:38:31 +0000 Puppet (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:38:33 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Sleeping for 5.0 seconds between tries
2015-12-16 16:38:38 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Exec try 2/50
2015-12-16 16:38:38 +0000 Exec[nailgun_upload_fixtures](provider=posix) (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:38:38 +0000 Puppet (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:38:39 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Sleeping for 5.0 seconds between tries
2015-12-16 16:38:44 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Exec try 3/50
...
2015-12-16 16:43:34 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Sleeping for 5.0 seconds between tries
2015-12-16 16:43:39 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Exec try 50/50
2015-12-16 16:43:39 +0000 Exec[nailgun_upload_fixtures](provider=posix) (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:43:39 +0000 Puppet (debug): Executing '/usr/bin/nailgun_fixtures'
2015-12-16 16:43:40 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (debug): Sleeping for 5.0 seconds between tries
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:39.702 DEBUG [7fb09ae1f740] (settings) Looking for settings.yaml package config using old style __file__
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:39.702 DEBUG [7fb09ae1f740] (settings) Trying to read config file /usr/lib/python2.7/site-packages/nailgun/settings.yaml
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:39.865 DEBUG [7fb09ae1f740] (settings) Trying to read config file /etc/nailgun/settings.yaml
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.691 INFO [7fb09ae1f740] (fixman) Fixture model 'release' with pk='1' already uploaded. Skipping
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.697 INFO [7fb09ae1f740] (fixman) Fixture model 'release' with pk='2' already uploaded. Skipping
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.698 INFO [7fb09ae1f740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixture
s/openstack.yaml
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.701 INFO [7fb09ae1f740] (fixman) Fixture model 'notification' with pk='1' already uploaded. Skipping
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.701 INFO [7fb09ae1f740] (fixman) Fixture has been uploaded from file: /usr/lib/python2.7/site-packages/nailgun/fixture
s/start_notification.json
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.706 INFO [7fb09ae1f740] (fixman) Fixture model 'network_group' with pk='1' already uploaded. Skipping
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.709 INFO [7fb09ae1f740] (fixman) Fixture model 'i_p_addr' with pk='1' already uploaded. Skipping
2015-12-16 16:43:45 +0000 /Stage[main]/Nailgun::Venv/Exec[nailgun_upload_fixtures]/returns (notice): 2015-12-16 16:43:40.711 ERROR [7fb09ae1f740] (fixman) Can't resolve foreign key 'network_group' for object '<class 'nailgun.db.sqlalchemy.
models.network.IPAddrRange'>'

http://paste.openstack.org/show/482106/

As you can see '/usr/bin/nailgun_fixtures' script fails to upload data to DB, because creation of IPAddrRange is not skipped. It happens because after changing some network parameters (e.g. public network range for some cluster) IPAddrRange with id = 1 (primary key) is removed.

Steps to reproduce:

1. Setup Fuel master
2. Create some environment
3. Change public network IP range (and save changes)
4. Restart nailgun container (dockerctl restart nailgun)

Expected result: container is restarted, there are no errors in puppet logs and puppet apply took less than 1 minute

Actual result: puppet apply took more than 5 minutes and its logs contain errors

[Ihor Kalnytskyi (ikalnytskyi)]

Weird. :( nailgun_fixtures must be run only once, during master node setup. Any further attempt must break one day something, since fixture maps directly to database, including primary key. That's why fixture is supposed to be *initial* database setup, and must be run only once.

[Stanislaw Bogatkin (sbogatkin)]

Ressigned to python team, as library cannot fix it by changing upload_fixtures call.

[Alexander Kislitsky (akislitsky)]

As possible solution we can save fuxtures application fact in the DB.
And as dirty workaround - if we have even one fixture applied - just exit from upload_fixtures method.

[Ihor Kalnytskyi (ikalnytskyi)]

No dirty workarounds. We have enough ones in Nailgun.

This is something library part should apply only once. I see no reasons why library can't do so. Please provide detailed description why it's impossible to do so.

[Bogdan Dobrelya (bogdando)]

Please elaborate, might reuploading fixtures -whilst it shall not- be destructive to the DB? If so, I'd raise this to the critical. The point is we don't test containers restarting much (please correct me if I'm wrong), so we probably have no data to triage this case w/o deep knowledge of Nailgun DB.

[Bogdan Dobrelya (bogdando)]

related QA bug https://bugs.launchpad.net/fuel/+bug/1527262

[Bogdan Dobrelya (bogdando)]

Puppet library of manifests has no knowledge of the first run of the master node controllers. There is no difference between first run vs just a run for CM cases, AFAIK. If we want this to be present, the backend shall provide this knowledge to the library. There is not another way to go

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260564

[Ihor Kalnytskyi (ikalnytskyi)]

Bogdan,

> If we want this to be present, the backend shall provide this knowledge to the library

Backend shall do nothing. Uploading fixture is a part of master node deployment, such as calling "syncdb". It MUST NOT be a part of Nailgun puppet inside. It MUST be executed once and that's it.

> There is not another way to go

There's is. Please move calling nailgux_fixtures to puppet manifests that deploys master node.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/260564
Reason: Decided to fix an issues with minimal changes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262018
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d3270d427b60c8d352bb2d78711e569ddc5f1903
Submitter: Jenkins
Branch: master

commit d3270d427b60c8d352bb2d78711e569ddc5f1903
Author: Alexander Kislitsky <email address hidden>
Date: Mon Dec 28 18:21:33 2015 +0300

    Uploading fixtures only for clear DB introduced

    We should upload fixtures only once at filling the empty DB.
    Applying fixtures on the production DB can mess the data.

    Change-Id: I88a90230e7898061f4ef0ed094f53a3fba4702fa
    Partial-Bug: #1526917
    Closes-Bug: #1528620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/268091

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268091
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=be76f73fd2444b1b970ecb196aea299b51a3e82c
Submitter: Jenkins
Branch: stable/8.0

commit be76f73fd2444b1b970ecb196aea299b51a3e82c
Author: Alexander Kislitsky <email address hidden>
Date: Mon Dec 28 18:21:33 2015 +0300

    Uploading fixtures only for clear DB introduced

    We should upload fixtures only once at filling the empty DB.
    Applying fixtures on the production DB can mess the data.

    Change-Id: I88a90230e7898061f4ef0ed094f53a3fba4702fa
    Partial-Bug: #1526917
    Closes-Bug: #1528620
    (cherry picked from commit d3270d427b60c8d352bb2d78711e569ddc5f1903)

[Alexander Kislitsky (akislitsky)]

Summary of discussion with Igor Kalintsky:

Fixtures should be applied only once - on the deployment stage, not in the container start process. It should be done on the fuel-library side.

Current fix is enough for 8.0 bug can be closed for this branch.

[Andrey Sledzinskiy (asledzinskiy)]

verified on 8.0-506

[Matthew Mosesohn (raytrac3r)]

We can use one of the config files or the package for nailgun as a basis to determine if we have already synchronized fixtures. In a normal situation, we shouldn't ever repeat the sync fixtures. Alternatively, we can make a file that indicates fixtures have been uploaded. This part seems quite low priority since the https://review.openstack.org/#/c/262018 seems to already fix this problem adequately.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276621

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276769

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276769
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d73af357ce11bc2f5b1d6c4bdd0f8f5d43c56bb0
Submitter: Jenkins
Branch: master

commit d73af357ce11bc2f5b1d6c4bdd0f8f5d43c56bb0
Author: Matthew Mosesohn <email address hidden>
Date: Fri Feb 5 18:55:55 2016 +0400

    Fix mysql hash lookup for deep hash merge

    hiera_hash('mysql_hash') only pulls in the
    mysql_hash from globals.yaml, ignoring the
    'mysql' hash that exists in plugins.

    This is part of a fix that will align all hash
    storage and lookups to not contain '_hash' suffix.

    Change-Id: Id74f6b31107f8939f185b3832c897d3dff7197f9
    Partial-Bug: #1526917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/277458

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277458
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=d008eaf5de51ba4b19f4b15baf11ae91fe51a713
Submitter: Jenkins
Branch: stable/8.0

commit d008eaf5de51ba4b19f4b15baf11ae91fe51a713
Author: Matthew Mosesohn <email address hidden>
Date: Fri Feb 5 18:55:55 2016 +0400

    Fix mysql hash lookup for deep hash merge

    hiera_hash('mysql_hash') only pulls in the
    mysql_hash from globals.yaml, ignoring the
    'mysql' hash that exists in plugins.

    This is part of a fix that will align all hash
    storage and lookups to not contain '_hash' suffix.

    Change-Id: Id74f6b31107f8939f185b3832c897d3dff7197f9
    Partial-Bug: #1526917
    (cherry picked from commit d73af357ce11bc2f5b1d6c4bdd0f8f5d43c56bb0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276621
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=291759df225105ea53b6557618670f60dd68fb1c
Submitter: Jenkins
Branch: master

commit 291759df225105ea53b6557618670f60dd68fb1c
Author: Matthew Mosesohn <email address hidden>
Date: Fri Feb 5 12:30:13 2016 +0400

    Run syncdb and nailgun_fixtures only on initial deploy

    nailgun_syncdb and nailgun_fixtures tasks are run only
    when nailgun settings.yaml is updated now, removing the
    chances of creating errors when the DB structure has
    changed since the initial deployment.

    Change-Id: I041ef2391e709b4963ad05c6a63ce2780aae2f74
    Closes-Bug: #1526917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/300038

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300038
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=207a762adf0d3501bb5af058eb44b6e730b88d28
Submitter: Jenkins
Branch: stable/7.0

commit 207a762adf0d3501bb5af058eb44b6e730b88d28
Author: Matthew Mosesohn <email address hidden>
Date: Fri Feb 5 18:55:55 2016 +0400

    Fix mysql hash lookup for deep hash merge

    hiera_hash('mysql_hash') only pulls in the
    mysql_hash from globals.yaml, ignoring the
    'mysql' hash that exists in plugins.

    This is part of a fix that will align all hash
    storage and lookups to not contain '_hash' suffix.

    Change-Id: Id74f6b31107f8939f185b3832c897d3dff7197f9
    Partial-Bug: #1526917
    (cherry picked from commit d73af357ce11bc2f5b1d6c4bdd0f8f5d43c56bb0)
    (cherry picked from commit d008eaf5de51ba4b19f4b15baf11ae91fe51a713)

[Matthew Mosesohn (raytrac3r)]

Correction: Commit https://review.openstack.org/300038 is for bug https://bugs.launchpad.net/fuel/+bug/1542307 and doesn't affect this bug.

[TatyanaGladysheva (tgladysheva)]

According to comment #25, bug moves to Confirmed status.

[Vitaly Sedelnik (vsedelnik)]

Won't Fix for 7.0-updates because of Won't Fix status for 8.0 (we need fix in 8.0 in order to backport it to 7.0)

[Dmitriy Kruglov (dkruglov)]

Containers were removed from master in MOS 9.0, so this bus is no longer actual.