Fuel for OpenStack

redeployment fails because of recreating user nova in mysql

Bug #1537325 reported by Gleb on 2016-01-23
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
​ Fuel for OpenStack
Fix Committed
Medium
Alexey Deryugin
Fuel for OpenStack 7.0-updates
Also affects project (?) Also affects distribution/package Nominate for series

Bug Description

{"build_id": "165", "build_number": "165", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "165", "build_number": "165", "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["experimental"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha": "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups": ["experimental"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release": "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}

I'm deploying MOS 7.0 with local ubuntu mirror. If some ubuntu packages are absent deployment fails.

After redeployment we get puppet error because manifest are trying to create user nova in mysql whereas this user was already created. It looks like manifest aren't idempotent .

It's commercial deployment and I can't provide you with diagnostic snapshot because of customers security rules.

That's the error from puppet log:

2016-01-23 01:47:50 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:47:50 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:12 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:12 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:32 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'
2016-01-23 01:48:32 +0000 /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::Mysql[nova]/Openstacklib::Db::Mysql::Host_access[nova_cntl-SV-001]/Database_user[nova@cntl-SV-001]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001' identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'nova'@'cntl-sv-001'

Ivan Ponomarev (ivanzipfer) wrote :

Can you provide the purged packages that we have step to reproduce

Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
tags: added: area-library
Maksim Malchuk (mmalchuk) on 2016-01-25
tags: added: customer-found
Ivan Ponomarev (ivanzipfer) on 2016-01-25
Changed in fuel:
status: New → Confirmed
Ivan Ponomarev (ivanzipfer) on 2016-01-25
Changed in fuel:
importance: Undecided → Medium
Gleb (gleb-q) wrote : Re: [Bug 1537325] Re: redeployment fails because of recreating user nova in mysql
Download full text (5.4 KiB)

Our deployment failed because of packages libnss3-tools and
libapache2-mod-fastcgi were absent in local mirror.

Probably you don't need to reproduce, just check manifests that creates
user nova. It tries to create new user even it already exists.

On Mon, Jan 25, 2016 at 9:48 PM, Ivan Ponomarev <email address hidden>
wrote:

> Can you provide the purged packages that we have step to reproduce
>
> ** Changed in: fuel
> Assignee: (unassigned) => Fuel Library Team (fuel-library)
>
> ** Tags added: area-library
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1537325
>
> Title:
> redeployment fails because of recreating user nova in mysql
>
> Status in Fuel for OpenStack:
> Confirmed
>
> Bug description:
> {"build_id": "165", "build_number": "165", "release_versions":
> {"2015.1.0-7.0": {"VERSION": {"build_id": "165", "build_number":
> "165", "api": "1.0", "fuel-library_sha":
> "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha":
> "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups":
> ["experimental"], "fuel-nailgun-agent_sha":
> "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version":
> "2015.1.0-7.0", "fuel-agent_sha":
> "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker",
> "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
> "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-
> ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release":
> "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}}},
> "auth_required": true, "api": "1.0", "fuel-library_sha":
> "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd", "nailgun_sha":
> "4162b0c15adb425b37608c787944d1983f543aa8", "feature_groups":
> ["experimental"], "fuel-nailgun-agent_sha":
> "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version":
> "2015.1.0-7.0", "fuel-agent_sha":
> "50e90af6e3d560e9085ff71d2950cfbcca91af67", "production": "docker",
> "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3",
> "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-
> ostf_sha": "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c", "release":
> "7.0", "fuelmain_sha": "a65d453215edb0284a2e4761be7a156bb5627677"}
>
>
> I'm deploying MOS 7.0 with local ubuntu mirror. If some ubuntu packages
> are absent deployment fails.
>
> After redeployment we get puppet error because manifest are trying to
> create user nova in mysql whereas this user was already created. It
> looks like manifest aren't idempotent .
>
> It's commercial deployment and I can't provide you with diagnostic
> snapshot because of customers security rules.
>
>
> That's the error from puppet log:
>
> 2016-01-23 01:47:50 +0000 Puppet (err): Execution of '/usr/bin/mysql
> --defaults-extra-file=/root/.my.cnf mysql -e create user 'nova'@'cntl-SV-001'
> identified by PASSWORD '*CD404B6942DAB5A6EF07255F2F935118419D131F''
> returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for
> 'nova'@'cntl-sv-001'
> 2016-01-23 01:47:50 +0000
> /Stage[main]/Nova::Db::Mysql/Openstacklib::Db::M...

Read more...

Ivan Ponomarev (ivanzipfer) on 2016-01-26
Changed in fuel:
milestone: none → 9.0
Michael Polenchuk (mpolenchuk) wrote :

not affected version 8.x+

Changed in fuel:
milestone: 9.0 → 7.0-updates
Julia Varlamova (jvarlamova) on 2016-02-18
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Julia Varlamova (jvarlamova)
Julia Varlamova (jvarlamova) wrote :

Unfortunately, I was unable to reproduce this bug in MOS 7.0 deployment (Ubuntu). When applying the manifest which creates user 'nova' in debug mode we can see that it checks if user 'nova' already exists before creation. It means that this manifest is idempotent. For now this bug seems to me to be invalid, but maybe I'm wrong? Maybe someone was able to reproduce this bug?

seroron (seroron) wrote :
Download full text (5.6 KiB)

I reproduced this bug in both MOS 7.0 and MOS 8.0 deployment (Ubuntu).

The following is MOS 8.0 log.
=======================
2016-03-06 01:50:40 ERR (/Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Openstacklib::Db::Mysql::Host_access[keystone_CTRNODE03]/Database_user[keystone@CTRNODE03]/ensure) change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'keystone'@'CTRNODE03' identified by PASSWORD '*XXXXXXXXXXXXXXXXXXXXXXXXX'' returned 1: ERROR 1396 (HY000) at line 1: Operation CREATE USER failed for 'keystone'@'CTRNODE03'
2016-03-06 01:50:40 ERR /usr/bin/puppet:8:in `<main>'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2016-03-06 01:50:40 ERR /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:120:in `block in apply_catalog'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:168:in `apply'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:112:in `as_logging_destination'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'
2016-03-06 01:50:40 ERR /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:169:in `block in apply'
2016-03-06 01:50...

Read more...

Julia Varlamova (jvarlamova) on 2016-03-18
Changed in fuel:
assignee: Julia Varlamova (jvarlamova) → nobody
Maciej Relewicz (rlu) on 2016-03-18
Changed in fuel:
assignee: nobody → Fuel Library Team (fuel-library)
Max Stepanov (t-max-z) wrote :

Puppet resource is not idempotent. This bug is due to incorrect checking if user is already created.
Both those cases happened because hostname contains uppercase characters. When "CREATE USER" query is processed by MySQL hostname will be converted to lower case. But when puppet checks if user was already created it uses query like so:

select '1' from user where CONCAT(user, '@', host) = 'cinder@HOSTNAME1';

Obviously it will not match the existing records. So puppet tries to create a "new" user and fails because mysql will convert the hostname to lower case and there is already a record like that.

Temporary solution is to avoid upper case letters in hostname for nodes.

Bogdan Dobrelya (bogdando) on 2016-03-24
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → MOS Puppet Team (mos-puppet)
tags: added: area-mos
removed: area-library
Ivan Berezovskiy (iberezovskiy) on 2016-06-27
Changed in fuel:
assignee: MOS Puppet Team (mos-puppet) → Alexey Deryugin (velovec)
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/334783

Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to puppet-modules/puppet-openstacklib (mos-8.0)

Fix proposed to branch: mos-8.0
Change author: Alexey Deryugin <email address hidden>
Review: https://review.fuel-infra.org/22650

Changed in fuel:
status: Confirmed → In Progress
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to puppet-modules/puppet-openstacklib (mos-8.0)

Reviewed: https://review.fuel-infra.org/22650
Submitter: Ivan Berezovskiy <email address hidden>
Branch: mos-8.0

Commit: 2b094fc75255a4be4a98440f7df551da946c26dc
Author: Alexey Deryugin <email address hidden>
Date: Tue Jul 5 11:20:35 2016

Downcase all hostnames for MySQL user/db creation

Mysql puppet providers apply downcase method for hostnames, that
lead us to errors when we use hostnames with upper-case letters.

Change-Id: Ic1d73a3d5282ac71c4c16b71863c8507205a74f6
Closes-Bug: #1537325

Changed in fuel:
status: In Progress → Fix Committed
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Alexey Deryugin (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/334783

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.
Subscribing...

Other bug subscribers