Create and sync DB entities only at primary-controller

[Anastasia Palkina]

"build_id": "2014-06-16_00-31-15",
"mirantis": "yes",
"build_number": "255",
"ostf_sha": "67b61ed3788297fa5d985afec32498d8c0f812db",
"nailgun_sha": "984aa7a86487f1488c2f83c052904abd9f589b7f",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "6f355160366475d52050d7898a1080a95ecb9cbf",
"astute_sha": "17b1afa5f0dc8f4fca5ed4eb03ec566fbfb5ed19",
"release": "5.1",
"fuellib_sha": "99d74172887ab81d38132655d6e5d180e8726437"

1. Create new environment (Ubuntu, HA mode)
2. Choose GRE segmentation
3. Add 3 controller+cinder, compute
4. Start deployment. It was successful
5. But there are errors in puppet.log on second and third controller (node-15,16)

Mon Jun 16 16:17:36 +0000 2014 Puppet (err): Execution of '/usr/bin/keystone --os-token Z3nTUwQp --os-endpoint http://192.168.0.7:35357/v2.0/ user-role-remove --user-id ce40b9cdeffe44f98272e8edbff54661 --tenant-id 9c265d89f20046dabdb1da84f0c8ab06 --role-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)
Mon Jun 16 16:17:36 +0000 2014 /Stage[main]/Swift::Keystone::Auth/Keystone_user_role[swift@services]/roles (err): change from _member_admin to admin failed: Execution of '/usr/bin/keystone --os-token Z3nTUwQp --os-endpoint http://192.168.0.7:35357/v2.0/ user-role-remove --user-id ce40b9cdeffe44f98272e8edbff54661 --tenant-id 9c265d89f20046dabdb1da84f0c8ab06 --role-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)

This should not happen if the Nova::Db::Mysql/Nova::Db::Mysql::Host_access[Database_user] puppet module commands are only run on a single controller node. The fix for this is to ensure that the above Puppet module/command is only ever run on a single node.

Same goes for the rest of the {Keystone,Cinder,Glance,etc.}::Db::Mysql classes and for all db_sync commands.

[Dmitry Borodaenko (angdraug)]

The most recent commit that modified ordering of keystone_user_role resources in fuel-library:
https://review.openstack.org/100263

[Dmitry Borodaenko (angdraug)]

Review linked above (https://review.openstack.org/100263) is included in ISO #259, so it does not fix this bug. I think the reordering of resources done in that commit is what actually brought this bug to the surface.

[Sergey Vasilenko (xenolog)]

> The most recent commit that modified ordering of keystone_user_role resources in fuel-library: https://review.openstack.org/100263

This request will be merged after register this bug.

> 4. Start deployment. It was successful
> 5. But there are errors in puppet.log on second and third controller (node-15,16)

looks like we try to remove user on 2nd and 3d controller, but this user already removed while deploy on 1st.

[Anastasia Palkina (apalkina)]

Second controller - node-12
Third controller - node-13

[Sergey Vasilenko (xenolog)]

for last attached logs:

root@node-11:~# keystone user-list
+----------------------------------+---------+---------+-------------------+
| id | name | enabled | email |
+----------------------------------+---------+---------+-------------------+
| 4c40238975c14b0993ab61b8bd5f23d5 | admin | True | <email address hidden> |
| b17a57e3eb034d4abb8e268a62e6369f | cinder | True | cinder@localhost |
| 2a6d1013d7094aee97f8d7bb73dd02db | glance | True | glance@localhost |
| 70aea00c49d34a33ade5cf22afb2342e | heat | True | <email address hidden> |
| 47af1b7c70324aa09c21d3150fe7339b | neutron | True | neutron@localhost |
| 18244dd625e74cd5b96402b02d1a0ab3 | nova | True | nova@localhost |
| d479383093d248ac959d2aeab4305366 | swift | True | |
+----------------------------------+---------+---------+-------------------+

root@node-11:~# keystone tenant-list
+----------------------------------+----------+---------+
| id | name | enabled |
+----------------------------------+----------+---------+
| df2f8d0036e24c68932a5d4af915c941 | admin | True |
| 18ea7c1acad14118a8e73385c8da3e6d | services | True |
+----------------------------------+----------+---------+

root@node-11:~# keystone role-list
+----------------------------------+---------------+
| id | name |
+----------------------------------+---------------+
| dac05cb2216b497f85375d79ee1ecfc3 | Member |
| 1423cf9e85a0405b8fdf3a419e589aef | SwiftOperator |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
| 9b69d504a7bd4f09b86af359edd2dc7e | admin |
+----------------------------------+---------------+

[Bogdan Dobrelya (bogdando)]

Could be related http://lists.openstack.org/pipermail/openstack-dev/2014-June/038614.html

[Anastasia Palkina (apalkina)]

Reproduced on ISO #270 on CentOS
"build_id": "2014-06-25_00-31-14",
"mirantis": "yes",
"build_number": "270",
"ostf_sha": "4d2efa822344b6ca022ec4086b6f083c07d90e14",
"nailgun_sha": "eeb88eecafa11a200de8f169a29975506dda29b2",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "e1fe73e77b7a89a035540390f5a6f6e5c8fb3615",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "d204858549ce3e118935fb2a9ed8a907dd197bb5"

1. Create new environement (CentOS, HA mode)
2. Choose GRE segmentation
3. Add 3 controllers, 1 compute
4. Start deployment. It was successful
5. But there are errors on second and third controller (node-2,3):

2014-06-25 12:03:31 ERR

 (/Stage[main]/Swift::Keystone::Auth/Keystone_user_role[swift@services]/roles) change from _member_admin to admin failed: Execution of '/usr/bin/keystone --os-token AjZ31Ilj --os-endpoint http://192.168.0.4:35357/v2.0/ user-role-remove --user-id 1304450c1fd842c0afb96d82b2a64f41 --tenant-id 2783da7758cd4f6b95681af45934abac --role-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)

[Anastasia Palkina (apalkina)]

I reproduced the analogue bug for ISO #82, version 5.0.1

"build_id": "2014-07-01_15-24-44",
"mirantis": "yes",
"build_number": "82",
"ostf_sha": "d0fe60e0eba61685008b86d101f459fc2d3bb654",
"nailgun_sha": "63a852bc402c079083a8cd0896c44254a1adcdbc",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "29efe784697bf3030bfb2b6ef2c6bc1deb10b587",
"astute_sha": "644d279970df3daa5f5a2d2ccf8b4d22d53386ff",
"release": "5.0.1",
"fuellib_sha": "464e3b55637fc1aaefe71e41d1336a100d9abf7b"

1. Create new environment (CentOS, HA mode)
2. Choose nova-network, vlan manager
3. Add 4 controllers, 2 computes and 3 cinders
4. Start deployment. It was successful
5. There are errors on the first controller (node-3):
2014-07-01 16:11:32 ERR

 (/Stage[main]/Keystone::Roles::Admin/Keystone_user_role[admin@admin]/roles) change from _member_admin to admin failed: Execution of '/usr/bin/keystone --os-token iUtN48Co --os-endpoint http://192.168.0.3:35357/v2.0/ user-role-remove --user-id 12e97688e1254a27adc18134494fc7bb --tenant-id 1a46da0595f8437ea3f8bbae14322ebf --role-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Request to http://192.168.0.3:35357/v2.0/tenants/1a46da0595f8437ea3f8bbae14322ebf/users/12e97688e1254a27adc18134494fc7bb/roles/OS-KSADM/9fe2ff9ee4384b1894a90878d3e92bab timed out

[Bogdan Dobrelya (bogdando)]

From logs in #6 and ids in #7 there is a clear root cause could be seen, which are race conditions
caused by concurrent puppet applies at controllers :

At node-11, DELETE role _member_ ok for:
2014-06-20T13:14:43.220721, | 70aea00c49d34a33ade5cf22afb2342e | heat | True | <email address hidden> |
2014-06-20T13:15:08.737786, | 18244dd625e74cd5b96402b02d1a0ab3 | nova | True | nova@localhost |
2014-06-20T13:15:49.961279, | 47af1b7c70324aa09c21d3150fe7339b | neutron | True | neutron@localhost |

Next at node-13, DELETE role _member_ ok for:
2014-06-20T13:53:01.596854, | 4c40238975c14b0993ab61b8bd5f23d5 | admin | True | <email address hidden> |

Next at node-12, DELETE role _member_ FAIL (2014-06-20T13:57:17.589411) for:
2014-06-20T13:57:15.504315, | d479383093d248ac959d2aeab4305366 | swift | True | |
and same fail next at node-13, 2014-06-20T13:57:17.589411+01:00

Next, DELETE role _member_ ok for cinder user at node-13
2014-06-20T13:57:24.829047, | b17a57e3eb034d4abb8e268a62e6369f | cinder | True | cinder@localhost |
but at 2014-06-20T13:57:25.838060 failed at node-12, now for cinder - and so on...

The issue is not relevant anymore due to we reverted concurrent deployment feature

[Bogdan Dobrelya (bogdando)]

reproduced
{"build_id": "2014-08-22_10-44-21", "ostf_sha": "907f25f8fad39b177bf6a66fba9785afa7dd8008", "build_number": "468", "auth_required": true, "api": "1.0", "nailgun_sha": "e9dd053fce12908b5a5a404435ce2e8200450a23", "production": "docker", "fuelmain_sha": "5a7df58786db7962a1774b5be3611c4e7543015d", "astute_sha": "ac520b09525af4551e730b1c1f78170fefaf3cb8", "feature_groups": ["mirantis"], "release": "5.1", "fuellib_sha": "bddba1e854a6b0350e844a0baad50816d3cc8e28"}

2014-08-26T11:35:36.651517 node-8 ./node-8.test.domain.local/puppet-apply.log:2014-08-26T11:35:36.651517+01:00 notice: Finished catalog run in 932.3
7 seconds
2014-08-26T11:44:45.956818 node-9 ./node-9.test.domain.local/puppet-apply.log:2014-08-26T11:44:45.956818+01:00 err: Execution of '/usr/bin/keystone
--os-token tFUZ7ard --os-endpoint http://10.108.3.4:35357/v2.0/ user-role-remove --user-id da8aa06786d64cd5831dc45026e4cdfd --tenant-id 6951dbcb63ef4
4418cb0e41f2d332a37 --role-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e
92bab (HTTP 404)
2014-08-26T11:44:45.959873 node-9 ./node-9.test.domain.local/puppet-apply.log:2014-08-26T11:44:45.959873+01:00 err: (/Stage[main]/Glance::Keystone::
Auth/Keystone_user_role[glance@services]/roles) change from _member_admin to admin failed: Execution of '/usr/bin/keystone --os-token tFUZ7ard --os-e
ndpoint http://10.108.3.4:35357/v2.0/ user-role-remove --user-id da8aa06786d64cd5831dc45026e4cdfd --tenant-id 6951dbcb63ef44418cb0e41f2d332a37 --role
-id 9fe2ff9ee4384b1894a90878d3e92bab' returned 1: Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)
2014-08-26T11:44:46.145025 node-9 ./node-9.test.domain.local/keystone.log:2014-08-26T11:44:46.145025+01:00 debug: 2014-08-26 10:44:45.950 12721 WARN
ING keystone.common.wsgi [-] Cannot remove role that has not been granted, 9fe2ff9ee4384b1894a90878d3e92bab
2014-08-26T11:46:59.006154 node-10 ./node-10.test.domain.local/puppet-apply.log:2014-08-26T11:46:59.006154+01:00 notice: Finished catalog run in 657
.82 seconds

[Bogdan Dobrelya (bogdando)]

keystone user-list | grep da8aa06786d64cd5831dc45026e4cdfd
| da8aa06786d64cd5831dc45026e4cdfd | glance | True | glance@localhost |
keystone tenant-list | grep 6951dbcb63ef44418cb0e41f2d332a37
| 6951dbcb63ef44418cb0e41f2d332a37 | services | True |
keystone role-list | grep 9fe2ff9ee4384b1894a90878d3e92bab
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/116895

[Bogdan Dobrelya (bogdando)]

related bug https://bugs.launchpad.net/fuel/+bug/1361676 for db syncing

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/116906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/116906
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=43ed4ebdbf208d09a1de4f88bcdecb59a2ef2a03
Submitter: Jenkins
Branch: master

commit 43ed4ebdbf208d09a1de4f88bcdecb59a2ef2a03
Author: Vladimir Kuklin <email address hidden>
Date: Tue Aug 26 18:36:53 2014 +0400

    Fix user role removal idempotency

    Catch user role removal 404 HTTP error
    not only in roles setter but also
    in destroy method

    Change-Id: I46b6ebc550c959de75a717c86e11282c6b4f12cd
    Closes-bug: #1330875

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/116895
Reason: superseded by granular deployment tasks

[Bogdan Dobrelya (bogdando)]

Restored https://review.openstack.org/116895 patch.
There is also related upstream bug https://bugs.launchpad.net/puppet-openstack/+bug/1445000

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/116895

[Bogdan Dobrelya (bogdando)]

We have a db_sync option in modules after Kilo upstream sync, let's just configure this to do db sync only for primary-controller role

[Aleksandr Didenko (adidenko)]

Moved to 8.0 as agreed with developers and QA

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/254811

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254811
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a607d9ad7ac176ee2874c2204d94080804287166
Submitter: Jenkins
Branch: master

commit a607d9ad7ac176ee2874c2204d94080804287166
Author: Denis Egorenko <email address hidden>
Date: Tue Dec 8 17:55:56 2015 +0300

    Run db sync only on primary controller

    We should run db sync only on primary controllers. Currently
    upstream modules have posibility to specify should we run
    db sync or don't.

    Change-Id: If061c0f1b2706ec4fd88966b8620e5586d98b0b8
    Closes-bug: #1330875

[Timur Nurlygayanov (tnurlygayanov)]

It looks like all works fine on MOS 8.0 RC1, status changed to Fix Released.