[regression][Keystone] Keystone detached plugin cannot connect to Mysql

[Evgeny Sikachev]

User Impact:
Users can't deploy environments with detached keystone plugin. Deployment will fail, we don't know the workaround yet.

ENVIRONMENT: ISO 495 MOS 9.0

STEPS TO REPRODUCE:
1. Deploy master node
2. Install plugin detached plugins
   id | name | version | package_version | releases
   ---+-----------------+---------+-----------------+--------------------
   1 | detach-database | 1.1.0 | 3.0.0 | ubuntu (mitaka-9.0)
   2 | detach-keystone | 1.0.2 | 3.0.0 | ubuntu (mitaka-9.0)
   3 | detach-rabbitmq | 1.0.1 | 3.0.0 | ubuntu (mitaka-9.0)
3. Deploy env 3 controller/mongo, 1 standalone-keystone, 1 standalone-database, 1 standalone-rabbitmq, 2 cinder/compute

EXPECTED RESULT:
User can sucessfully deploy the environment.

ACTUAL RESULT:
Deployment with separate Keystone plugin failed. Error in puppets "Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2"

shutgun2 short-report: http://paste.openstack.org/show/521186/

[Evgeny Sikachev (esikachev)]

snapshot https://drive.google.com/file/d/0B8UU8DiYLmsWeVdKNUd2ZWpqdUE/view?usp=sharing

[Sergii Turivnyi (sturivnyi)]

This bug is blocker for QA acceptance

[Alexander Makarov (amakarov)]

Evgeny,

Can you please share the criteria you used to decide this is a keystone
issue?
You are not the first, so I think it's a system problem that leads to such
decisions.

On Wed, Jun 22, 2016 at 8:13 PM, Sergii Turivnyi <email address hidden>
wrote:

> ** Tags added: fuel-dev-tools
>
> --
> You received this bug notification because you are a member of MOS
> Keystone, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1595219
>
> Title:
> [Keystone] Keystone plugin cannot connect to Mysql
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/fuel/+bug/1595219/+subscriptions
>

--
Kind Regards,
Alexander Makarov,
Senior Software Developer,

Mirantis, Inc.
35b/3, Vorontsovskaya St., 109147, Moscow, Russia

Tel.: [phone]
Tel.: [phone]

Skype: MAKAPOB.AJIEKCAHDP

[Timur Nurlygayanov (tnurlygayanov)]

Aleksandr, we just wanted to get help in debugging from team who understand the context of KEystone requirements/deployment workflow.

It looks like fuel-library regression issue.

[Alex Schultz (alex-schultz)]

Can you please provide the exact environment setup that this fails with? The snapshot does not contain all the logs so we'll need to be able to reproduce it.

[Alex Schultz (alex-schultz)]

Also it should be noted that we test this in swarm, which is currently passing so we need more details as to how this was configured.

[Alex Schultz (alex-schultz)]

I got access to an environment and it looks like the keystone/db task ran before all the database tasks were fully run so the database was not up yet when the task was run.

The detached-db database task started at
2016-06-22T16:03:28.400247+00:00 notice: (Scope(Class[Osnailyfacter::Database::Database])) MODULAR: database/database.pp
but does not complete until
2016-06-22T16:04:37.059760+00:00 notice: Finished catalog run in 64.94 seconds

Meanwhile the keystone/db task was run at...
2016-06-22T16:03:37.869731+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp
2016-06-22T16:03:47.721773+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp
2016-06-22T16:03:52.346289+00:00 notice: (Scope(Class[Openstack_tasks::Keystone::Db])) MODULAR: keystone/db.pp

so all retries occurred before the database was run.

[Alex Schultz (alex-schultz)]

RCA: detached keystone keystone-db task does not properly wait for all database tasks before attempting to to use the database. It only waited for the database tasks on the controllers so when this paired with the detached-database plugin the keystone-db task may run before the database is actually setup. As part of Bug 1591256, we updated the core keystone-db task to wait for all the database tasks before starting and so this needs to get ported to the plugin as well. See https://github.com/openstack/fuel-library/commit/bedd29639a500038a4d9f864f1d3d5a6e50345cc

ETA: 6/24/16

[Alex Schultz (alex-schultz)]

master: https://review.openstack.org/#/c/332969/
stable/mitaka: https://review.openstack.org/#/c/332970/

[Andrey Maximov (maximov)]

Marking non-release since it doesn't affect core fuel functionality

[Sergii Turivnyi (sturivnyi)]

can reproduce
Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

[Sergii Turivnyi (sturivnyi)]

Steps to reproduce
1. git clone https://github.com/stackforge/fuel-plugin-detach-keystone
2. git fetch https://git.openstack.org/openstack/fuel-plugin-detach-keystone refs/changes/70/332970/1 && git checkout FETCH_HEAD
3. fpb --build fuel-plugin-detach-keystone
4. copy plugin to the master-node
5. Create cluster
6. Deploy cluster with plugin

Actual result:
Could not prefetch mysql_grant provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/333364

[Alex Schultz (alex-schultz)]

@Sergii, Please try recreating the described issue from the bug with the plugin generated from the change. You can download the rpm from https://plugin-ci.fuel-infra.org/job/9.0.fuel-plugin.detach-keystone.build/4/artifact/plugin/detach-keystone-1.0-1.0.2-1.noarch.rpm

I wrote a fuel-qa test that describes an environment from the original report as well, https://review.openstack.org/#/c/333364/

I was unable to reproduce this error with these, I'm continuing to try and reproduce it.

[Alex Schultz (alex-schultz)]

So I took the bug conditions and created a fuel-qa test case (https://review.openstack.org/#/c/333364/) for this scenario. I then ran it using the existing detached plugins with fuel-qa and was able to reproduce the failure. The fuel-qa log can be viewed here: http://paste.openstack.org/show/521760/

I then took the rpm from the plugin-ci from https://review.openstack.org/#/c/332970 and ran it through the same test case about 4 times and was unable to reproduce the issue. The log can be viewed here, http://paste.openstack.org/show/521761/

I believe the proposed detached-keystone fixes address the issue and we now have an automated way to test this scenario.

[Sergey Shevorakov (sshevorakov)]

Moved from Crit to High after discussions with release, dev and product managers.

[Nastya Urlapova (aurlapova)]

Since we don't support detached keystone plugin in 9.1 release the issue moved to "Fix Released"

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/333364