galeracheck is broken in multirack envs

Bug #1498554 reported by Aleksandr Didenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Sergey Vasilenko
7.0.x
Won't Fix
High
Fuel Library (Deprecated)
8.0.x
Fix Released
High
Sergey Vasilenko

Bug Description

When we use multirack deployment and controllers are not in the 'default' nodegroup, then our deployment will fail on primary controller role:

2015-09-21 23:35:29 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf mysql -e create user 'keystone'@'127.0.0.1' identified by PASSWORD '*767F984986E0D70CC01CBCDC5AB2AAE69F0CF02F'' returned 1: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

Haproxy status:

mysqld FRONTEND Status: OPEN Sessions: 0 Rate: 0
mysqld node-2 Status: DOWN/SOCKERRSessions: 0 Rate: 0
mysqld node-4 Status: DOWN/L4CON Sessions: 0 Rate: 0
mysqld node-5 Status: DOWN/L4CON Sessions: 0 Rate: 0
mysqld BACKEND Status: DOWN Sessions: 0 Rate: 0

The problem is caused by restrictions in /etc/xinetd.d/galeracheck:

only_from = 127.0.0.1 240.0.0.2 10.145.2.0/24

Where 10.145.2.0/24 is management network of default group. It's configured here:

https://github.com/stackforge/fuel-library/blob/stable/7.0/deployment/puppet/osnailyfacter/modular/database/database.pp#L108

So we need to add here management networks from all nodegroups to fix this issue.

How to reproduce:
1) Deploy env with at least two nodegroups with no L3 connectivity between them, with separate admin, management, storage, private networks
2) Create new env with neutron vlan/gre, add new nodegroup
3) Add 1+ controllers from NON-default nodegroup, 1+ compute from default nodegroup
4) Deploy changes

Expected result:
Deployment is successful

Actual result:
Deployment fails on primary controller

tags: added: customer-found
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/226499

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/226499
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=f9dadee0d2961244f8e1a83d84e4110d5e60cafb
Submitter: Jenkins
Branch: master

commit f9dadee0d2961244f8e1a83d84e4110d5e60cafb
Author: Sergey Vasilenko <email address hidden>
Date: Tue Sep 22 13:36:26 2015 -0500

    Use function get_routable_networks_for_network_role() for

    fetch all networks, accessible from interface, mapped by
    network role. This lookup doesn't honor default gateway,
    only specifyed routes valued.

    Change-Id: I90d70fd32dcc3cb2a3372febf1e6dade53f85615
    Closes-Bug: #1498554

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Aleksandr Didenko (adidenko) wrote :

We should not backport fix for this bug to 7.0 MU. It's not the only problem of multirack deployments with controllers in non-default nodegroup, so backporting this single fix to 7.0 won't make them work. Other required fixes are still in development.

no longer affects: fuel/6.1.x
Dmitry Pyzhov (dpyzhov)
tags: added: area-library
tags: added: on-verification
Revision history for this message
ElenaRossokhina (esolomina) wrote :

Verified using initial scenario
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "517"
  build_id: "517"
  fuel-nailgun_sha: "ebede8c0efab72b6e80735eb3a8161f953d84b83"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "601a0378f3e7f42ee2bc8f3d5f54fc681b6c5102"
  fuel-ostf_sha: "5fe41945c2a49f26c849df1fd46329f6db1ab6b0"
  fuel-mirror_sha: "6f8e1a005446eb49981fe93bd9a67ca944c0a763"
  fuelmenu_sha: "12227354aec1d38f7f51042df64cca59fa7a95f1"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "a365f05b903368225da3fea9aa42afc1d50dc9b4"

Deployment is OK, /etc/xinetd.d/galeracheck contains all management networks

tags: removed: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
tags: added: wontfix-risky
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.