Fuel for OpenStack

Deployment process of openstack on the second controller of nsx's machine always fails due to different reasons.

Series 5.1.x
Bug #1368728

Bug #1368728 reported by Tatyana Dubyk on 2014-09-12

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	Critical	Stepan Rogov	Fuel for OpenStack 5.1
5.1.x	Fix Committed	Critical	Stepan Rogov	Fuel for OpenStack 5.1
6.0.x	Fix Released	Critical	Andrey Danin	Fuel for OpenStack 6.0

Bug Description

I've checked this bug on different configurations of env in ha mode.
After each fallen deploy I've reseted snapshot on nsx machine and cleared previous envs.

==============nsx settings===========================
Fuel UI-> Settings -> VMware NSX

NSX_USERNAME='admin'
NSX_PASSWORD= 'admin'
NSX_TRANSPORT_ZONE='0f12c1f7-5605-46b6-aa3f-4bf50bde47e5'
L3_SERVICE_UUID='aa153851-bd8e-4852-9772-d98a431849e0'
NSX_CONTROLLER_ENDPOINT='172.16.1.253'
URL_TO_NSX_BITS='http://10.20.204.1'

NSX network -> management -> untag and set 172.16.1.0/24 on eth2
=====================================================
Configuration 1:
===================================================
steps to reproduce:
1.set up lab on nsx machine
2.create env and start deploy:
   OS: Ubuntu (HA mode)
   hypervisor: QEMU
   roles: 3 controllers, 1 compute, 1 cinder

3. check that deployment of openstack on 1st controller is finished successfully
4. check that deployment of openstack on 2nd controller is failed

Expected result: Deployment process of openstack on each of nodes will be finished successfully
Actual result: deployment of openstack on 2nd controller is failed due to reason: Could not find init script or upstart conf file for 'glance-registry'

------------Logs------------------------------------

root@node-12:~# less /var/log/puppet.log

Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (notice): Dependency Service[mysql-service] has failure
s: true
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (warning): Skipping because of failed dependencies
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.conf in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.conf in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.sh in /etc/init
Thu Sep 11 19:45:49 +0000 2014 Service[glance-registry](provider=upstart) (debug): Could not find glance-registry.sh in /etc/init.d
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (err): Failed to call refresh: Could not find init script or upstart conf file for 'glance-registry'
Thu Sep 11 19:45:49 +0000 2014 /Stage[main]/Glance::Registry/Service[glance-registry] (err): Could not find init script or upstart conf file for 'glance-registry'

Glance package is not installed due to some reason.

root@node-12:~# aptitude show glance-registry
Package: glance-registry
State: not installed
Version: 1:2014.1.1-fuel5.1~mira10
Priority: extra
Section: python
Maintainer: Ubuntu OpenStack <email address hidden>
Architecture: all
Uncompressed Size: 21.5 k
Depends: glance-common (= 1:2014.1.1-fuel5.1~mira10), upstart-job, python
Recommends: python-glanceclient
Replaces: glance (< 2012.1~e1~20110919.1021-0ubuntu2), glance (< 2012.1~e1~20110919.1021-0ubuntu2)
Description: OpenStack Image Registry and Delivery Service - Registry
The Glance project provides an image registration, discovery and delivery service. These services may be used as stand-along services, and
they may also be used by Nova to deliver images from object stores, such as OpenStack's Swift service, to Nova's compute nodes.

This package contains the glance registry server.
Homepage: http://launchpad.net/glance

=================================================
Configuration 2:
=================================================
steps to reproduce:
1.set up lab on nsx machine
2.create env and start deploy:
   OS: Ubuntu (HA mode)
   hypervisor: QEMU
   roles: 2 controllers, 1 compute

3. check that deployment of openstack on 1st controller is finished successfully
4. check that deployment of openstack on 2nd controller is failed

Expected result: Deployment process of openstack on each of nodes will be finished successfully
Actual result:deployment of openstack on 2nd controller is failed due to reason: MySQL is not running

-----------------Logs------------------------------------
Trace from HC logs on node-7 (controller) (/Stage[main]/Galera/Service[mysql-service]/ensure) change from stopped to running failed: execution expired

Trace from node-7 : less /var/log less mysqld.log
<30>Sep 11 14:25:17 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:22 node-7 mysql-wss[2680]: INFO: MySQL is not running
<27>Sep 11 14:25:22 node-7 mysql-wss[2680]: ERROR: GTID have wrong format: :
<27>Sep 11 14:25:22 node-7 mysql-wss[2680]: ERROR: Wrong GTID, not updating gtid attribute
<30>Sep 11 14:25:22 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 2 retries left
<30>Sep 11 14:25:27 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 1 retries left
<30>Sep 11 14:25:32 node-7 mysql-wss[2680]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:37 node-7 mysql-wss[2680]: INFO: MySQL is not running
<30>Sep 11 14:25:37 node-7 mysql-wss[2913]: INFO: PIDFile /var/run/mysqld/mysqld.pid of MySQL server not found. Sleeping for 5 seconds. 0 retries left
<30>Sep 11 14:25:42 node-7 mysql-wss[2913]: INFO: MySQL is not running
<27>Sep 11 14:25:42 node-7 mysql-wss[2913]: ERROR: Directory /var/run/mysqld is not writable by mysql
<30>Sep 11 14:25:42 node-7 mysql-wss[2976]: INFO: MySQL is not running

no one service not available on node-7

and after manually restart and stop of mysql service too.

See original description

Tags:

Revision history for this message

Tatyana Dubyk (tdubyk) wrote on 2014-09-12:

Snapshot of configuration 1 Edit (101.9 MiB, application/x-tar)

Tatyana Dubyk (tdubyk) on 2014-09-12

description:

updated

Andrey Danin (gcon-monolake) on 2014-09-12

Changed in fuel:
importance:	Undecided → High

Revision history for this message

Tatyana Dubyk (tdubyk) wrote on 2014-09-12:

api: '1.0'
astute_sha: b622d9b36dbdd1e03b282b9ee5b7435ba649e711
auth_required: true
build_id: 2014-09-10_00-01-12
build_number: '506'
feature_groups:
- mirantis
fuellib_sha: e3f947cc4142210499a282bc5f183c333552fa23
fuelmain_sha: ca1b4839a70a10041f8eaf8b9ac995c8b0d4521a
nailgun_sha: 82091e0d61f252619a0842d0f8debb6b602a61fe
ostf_sha: 1de6ed1c0b72f6687ffb4bebc2c939b135a88e34
production: docker
release: '5.1'
release_versions:
  2014.1.1-5.1:
    VERSION:
      api: '1.0'
      astute_sha: b622d9b36dbdd1e03b282b9ee5b7435ba649e711
      build_id: 2014-09-10_00-01-12
      build_number: '506'
      feature_groups:
      - mirantis
      fuellib_sha: e3f947cc4142210499a282bc5f183c333552fa23
      fuelmain_sha: ca1b4839a70a10041f8eaf8b9ac995c8b0d4521a
      nailgun_sha: 82091e0d61f252619a0842d0f8debb6b602a61fe
      ostf_sha: 1de6ed1c0b72f6687ffb4bebc2c939b135a88e34
      production: docker
      release: '5.1'

Revision history for this message

Tatyana Dubyk (tdubyk) wrote on 2014-09-12:

nsx connector type: GRE

Evgeniya Shumakher (eshumakher) on 2014-09-12

Changed in fuel:
assignee:	nobody → Fuel Partner Integration Team (fuel-partner)

Stepan Rogov (srogov) on 2014-09-15

Changed in fuel:
assignee:	Fuel Partner Integration Team (fuel-partner) → Stepan Rogov (srogov)

Stepan Rogov (srogov) on 2014-09-16

Changed in fuel:
status:	New → Confirmed

Revision history for this message

Stepan Rogov (srogov) wrote on 2014-09-16:

The bug occurs because corosync can not run mysql.
Broke check resources before starting ("insufficient privileges").
This is because this test is carried out from the shell ("/bin/sh"). In ubuntu is a symlink to dash.
But in the file /etc/profile.d/nsx-alias.sh from the package nicira-ovs-hypervisor-node uses bash-syntax, though there specified interpreter /bin/sh.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-16: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/121915

Changed in fuel:
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-16: Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/121997

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-09-16: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/121915
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cd540218f53b933d8d5dd3c1aa347d60876f12d
Submitter: Jenkins
Branch: master

commit 5cd540218f53b933d8d5dd3c1aa347d60876f12d
Author: Stepan Rogov <email address hidden>
Date: Tue Sep 16 20:00:30 2014 +0400

Added file /etc/profile.d/nsx-alias.sh deletion in Ubuntu HA + NSX deploy

    Delete this file, because it uses bash-syntax for functions defenitions,
    but specified interpreter is /bin/sh and in Ubuntu /bin/sh->/bin/dash
    When corosync checks it resources, it uses /bin/sh which is equivalent
    /bin/dash in Ubuntu.
    Accordingly, corosync can not launch mysql.

Change-Id: I950ae6ad40c2c43c410cc0af98ff3b08b52a0260
Closes-bug: #1368728