redeployment fails because of recreating user nova in mysql

[Gleb]

{"build_id": "165", "build_number": "165", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "165", "build_number": "165", "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["experimental"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["experimental"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}

I'm deploying MOS 7.0 with local ubuntu mirror. If some ubuntu packages are absent deployment fails.

After redeployment we get puppet error because manifest are trying to create user nova in mysql whereas this user was already created. It looks like manifest aren't idempotent .

It's commercial deployment and I can't provide you with diagnostic snapshot because of customers security rules.

That's the error from puppet log:

2016-01-23 01:47:50 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:47:50 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:12 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:12 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:32 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:32 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'

[Ivan Ponomarev (ivanzipfer)]

Can you provide the purged packages that we have step to reproduce

[Gleb (gleb-q)]

Our deployment failed because of packages libnss3-tools and
libapache2-mod-fastcgi were absent in local mirror.

Probably you don't need to reproduce, just check manifests that creates
user nova. It tries to create new user even it already exists.

On Mon, Jan 25, 2016 at 9:48 PM, Ivan Ponomarev <email address hidden>
wrote:

> Can you provide the purged packages that we have step to reproduce
>
> ** Changed in: fuel
> Assignee: (unassigned) => Fuel Library Team (fuel-library)
>
> ** Tags added: area-library
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1537325
>
> Title:
> redeployment fails because of recreating user nova in mysql
>
> Status in Fuel for OpenStack:
> Confirmed
>
> Bug description:
> {"build_id": "165", "build_number": "165", "release_versions":
> {"2015.1.0-7.0": {"VERSION": {"build_id": "165", "build_number":
> "165", "api": "1.0", "fuel-library_sha":
> "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha":
> "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups":
> ["experimental"], "fuel-nailgun-agent_sha":
> "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version":
> "2015.1.0-7.0", "fuel-agent_sha":
> "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker",
> "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
> "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-
> ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release":
> "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}},
> "auth_required": true, "api": "1.0", "fuel-library_sha":
> "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha":
> "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups":
> ["experimental"], "fuel-nailgun-agent_sha":
> "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version":
> "2015.1.0-7.0", "fuel-agent_sha":
> "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker",
> "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
> "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-
> ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release":
> "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}
>
>
> I'm deploying MOS 7.0 with local ubuntu mirror. If some ubuntu packages
> are absent deployment fails.
>
> After redeployment we get puppet error because manifest are trying to
> create user nova in mysql whereas this user was already created. It
> looks like manifest aren't idempotent .
>
> It's commercial deployment and I can't provide you with diagnostic
> snapshot because of customers security rules.
>
>
> That's the error from puppet log:
>
> 2016-01-23 01:47:50 +0000 Puppet (err): Execution of '/usr/bin/mysql
> --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001'
> identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F''
> returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for
> 'nova'@'cntl-sv-001'
> 2016-01-23 01:47:50 +0000
> /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::M...

[Michael Polenchuk (mpolenchuk)]

not affected version 8.x+

[Julia Varlamova (jvarlamova)]

Unfortunately, I was unable to reproduce this bug in MOS 7.0 deployment (Ubuntu). When applying the manifest which creates user 'nova' in debug mode we can see that it checks if user 'nova' already exists before creation. It means that this manifest is idempotent. For now this bug seems to me to be invalid, but maybe I'm wrong? Maybe someone was able to reproduce this bug?

[seroron (seroron)]

I reproduced this bug in both MOS 7.0 and MOS 8.0 deployment (Ubuntu).

The following is MOS 8.0 log.
=======================
2016-03-06 01:50:40 ERR (/Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_CTRNODE03]/Database_user[keystone@CTRNODE03]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'keystone'@'CTRNODE03' identified by PASSWORD '*XXXXXXXXXXXXXXXXXXXXXXXXX'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'keystone'@'CTRNODE03'
2016-03-06 01:50:40 ERR /usr/bin/puppet:8:in `<main>'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2016-03-06 01:50:40 ERR /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
2016-03-06 01:50...

[Max Stepanov (t-max-z)]

Puppet resource is not idempotent. This bug is due to incorrect checking if user is already created.
Both those cases happened because hostname contains uppercase characters. When "CREATE USER" query is processed by MySQL hostname will be converted to lower case. But when puppet checks if user was already created it uses query like so:

select '1' from user where CONCAT(user, '@', host) = 'cinder@HOSTNAME1';

Obviously it will not match the existing records. So puppet tries to create a "new" user and fails because mysql will convert the hostname to lower case and there is already a record like that.

Temporary solution is to avoid upper case letters in hostname for nodes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/334783

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mos-8.0
Change author: Alexey Deryugin <email address hidden>
Review: https://review.fuel-infra.org/22650

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/22650
Submitter: Ivan Berezovskiy <email address hidden>
Branch: mos-8.0

Commit: 2b094fc75255a4be4a98440f7df551da946c26dc
Author: Alexey Deryugin <email address hidden>
Date: Tue Jul 5 11:20:35 2016

Downcase all hostnames for MySQL user/db creation

Mysql puppet providers apply downcase method for hostnames, that
lead us to errors when we use hostnames with upper-case letters.

Change-Id: Ic1d73a3d5282ac71c4c16b71863c8507205a74f6
Closes-Bug: #1537325

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexey Deryugin (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/334783