[Partition preservation] mysql is trying to use 'lost+found' database

[Ivan Ponomarev]

After reinstallation controller mysql defines and adds preserved partition with databases and tries to add 'lost+found' directory
The result is:
[ERROR] Invalid (old?) table or database name 'lost+found'
This happen because /var/lib/mysql is ext4 partition where file system created system directory 'lost+found'
Cluster deployed normally only error messages in logs.

The possible solution is additional directory where mysql should be located for example:
/var/lib/mysql/mysql_databases/

or changing format for this partition to xfs (I prefer this solution)

Step to reproduce:
1) Deploy standard openstack cluster
2) reinstall one controller node (provision, deploy using fuel CLI)
3) look on mysql logs

 VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "920"
  build_id: "2015-08-11_14-18-42"
  nailgun_sha: "8232ae9175a9e92ae8e0899834401ad81c10b30c"
  python-fuelclient_sha: "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc"
  fuel-library_sha: "fac1d453885e5f808d5b295830a9dc2b93af54b0"
  fuel-ostf_sha: "c7f745431aa3c147f2491c865e029e0ffea91c47"
  fuelmain_sha: "826387d1a05a40e4849433598442024bb1e3b7c7"

[Alexander Gordeev (a-gordeev)]

> tries to add 'lost+found' directory

any chance to configure mysql somehow to skip that directory?

[Alexander Gordeev (a-gordeev)]

> changing format for this partition to xfs (I prefer this solution based on our HA realization)

how is it related to a filesystem on partition and HA? It's not clear, please elaborate. As it looks like misconfiguration of mysql and seems to have nothing to do with the partitioning/provisioning stuff.

Also, it's not clear what happened with cluster besides throwing few error messages in mysql logs.
Was cluster deployed successfully? Did this error prevent mysql service from starting? Tell us more.

Please explain how does this issue affects deployment cluster. It is needed to figure out the importance of the bug.

Marking bug report as Incomplete, waiting for reply from the reporter. Thanks.

Looks like mysql should be started with '--ignore-db-dir=lost+found', but maybe i'm missing something.

Feel free to assign to fuel-library once you've done with all the questions.

[Ivan Ponomarev (ivanzipfer)]

Just errors in log. Cluster is working normally with it

[Roman Prykhodchenko (romcheg)]

This bug does not qualify to have high priority. Setting to Medium and moving to 8.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/227787

[Polina Petriuk (ppetriuk)]

This will also impact the sync of MySQL in HA when the new controller node is added or when one of the controllers fails and needs to be re-synced.

The following error is observed in mysql logs:
mysql_log_node-07.log:<27>Sep 22 15:55:27 node-07 mysqld: WSREP_SST: [ERROR] xtrabackup process ended without creating '/var/lib/mysql//xtrabackup_galera_info' (20150922 15:55:27.198)

On the Donor MySQL /var/lib/mysql/innobackup.backup.log will complaint - "mysql could not read from "lost+found" folder" since it's have root:root permissions.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/227787
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=fb100ee828d8c0262a8343155d307fba61b5f30b
Submitter: Jenkins
Branch: master

commit fb100ee828d8c0262a8343155d307fba61b5f30b
Author: Bartłomiej Piotrowski <email address hidden>
Date: Fri Sep 25 13:30:14 2015 +0200

    Make MySQL ignore lost+found directory in its datadir

    This commit introduces optional ignore_db_dirs parameter that can be passed to
    mysql::server. It is expected to be an array and is empty by default.

    Change-Id: Ic8815b89fea41182692f9f8fafd07bbe6ca15af6
    Closes-Bug: 1484552

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/231559

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/231559
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=eab5589b871baff978f00bc908dc5cb2dd8c854b
Submitter: Jenkins
Branch: stable/7.0

commit eab5589b871baff978f00bc908dc5cb2dd8c854b
Author: Bartłomiej Piotrowski <email address hidden>
Date: Fri Sep 25 13:30:14 2015 +0200

    Make MySQL ignore lost+found directory in its datadir

    This commit introduces optional ignore_db_dirs parameter that can be passed to
    mysql::server. It is expected to be an array and is empty by default.

    The backport also adds mounts fact that has been introduced in
    I8a2cb4bd8d0b6070400d13e25d2310f4777b9faf.

    Cherry-picked from: fb100ee828d8c0262a8343155d307fba61b5f30b
    Change-Id: Ic8815b89fea41182692f9f8fafd07bbe6ca15af6
    Closes-Bug: 1484552

[Roman Rufanov (rrufanov)]

Customer found on 6.1 - need backport.

[Alexander Zatserklyany (zatserklyany)]

Fix released on 7.0 mu1

[Alexey Stupnikov (astupnikov)]

Setting this bug to Invalid for MOS 6.1, since MOS6.1 doesn't allow to configure separate LV for MySQL database on controller nodes (it is hardcoded in fuel-web).

[Veronica Krayneva (vkrayneva)]

Verified on #491 iso.

[Vladimir Kuklin (vkuklin)]

reopening after seeing this with current master of fuel library (SHA 46ddfa166d451cb8826c7ea3c88a55abff836c06)

It seems, there is no $::mounts fact anymore, so ignore_db_dirs is not set

[Alex Schultz (alex-schultz)]

The fact is still there, https://github.com/openstack/fuel-library/blob/master/deployment/puppet/osnailyfacter/lib/facter/mounts.rb

Perhaps there's another issue?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/282793

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/282793
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1a826dd98b46d648f8d7055be52678542c14d6c8
Submitter: Jenkins
Branch: master

commit 1a826dd98b46d648f8d7055be52678542c14d6c8
Author: Maksim Malchuk <email address hidden>
Date: Sun Feb 21 01:33:23 2016 +0300

    Make MySQL ignore lost+found directory in its datadir

    This commit uses the correct way to provide configuration values to
    ignore directory names. To specify more than one name, the option
    'ignore-db-dir' should be used multiple times, once for each name.

    Change-Id: I0264cebcd5ebc5a1c9efe70505d79ca31c271fc9
    Closes-Bug: #1484552

[Aleksey Zvyagintsev (azvyagintsev)]

reproduced on 7.0 + updates
#
cat /var/log/mysql/error.log |grep -i lost
2016-03-02 11:39:57 28642 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-02 11:40:08 28642 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-02 11:40:20 28642 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-02 11:40:32 28642 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-02 11:40:40 28642 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-02 11:40:49 28642 [ERROR] Invalid (old?) table or database name 'lost+found'

#
cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "301"
  build_id: "301"
  nailgun_sha: "4162b0c15adb425b37608c787944d1983f543aa8"
  python-fuelclient_sha: "486bde57cda1badb68f915f66c61b544108606f3"
  fuel-agent_sha: "50e90af6e3d560e9085ff71d2950cfbcca91af67"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "6c5b73f93e24cc781c809db9159927655ced5012"
  fuel-library_sha: "5d50055aeca1dd0dc53b43825dc4c8f7780be9dd"
  fuel-ostf_sha: "2cd967dccd66cfc3a0abd6af9f31e5b4d150a11c"
  fuelmain_sha: "a65d453215edb0284a2e4761be7a156bb5627677"
#
[root@nailgun ~]# rpm -qa |grep fuel
fuelmenu-7.0.0-7670.1.git4162b0c.noarch
fuel-bootstrap-image-7.0.0-1.noarch
fuel-bootstrap-image-builder-7.0.0-1.noarch
python-fuelclient-7.0.0-201.1.git486bde5.noarch
fuel-docker-images-7.0.0-1.x86_64
fuel-target-centos-images6.6-7.0.0-1.noarch
fuel-createmirror-7.0-1.mos42.noarch
fuel-misc-7.0.0-7233.1.git5d50055.noarch
fuel-migrate-7.0.0-7233.1.git5d50055.noarch
fuel-provisioning-scripts-7.0.0-7670.1.git4162b0c.noarch
fuel-dockerctl-7.0.0-7233.1.git5d50055.noarch
fuel-library7.0-7.0.0-7233.1.git5d50055.noarch
fuel-notify-7.0.0-7233.1.git5d50055.noarch
fuel-7.0.0-6110.1.gita65d453.noarch
fuel-package-updates-7.0.0-7670.1.git4162b0c.noarch

[Maksim Malchuk (mmalchuk)]

also, reproduced on 8.0 release:

root@node-1:~# grep -i lost /var/log/mysql/error.log
2016-03-04 15:56:49 4033 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-04 15:56:55 4033 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-04 15:57:02 4033 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-04 16:05:47 4033 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-04 16:05:55 4033 [ERROR] Invalid (old?) table or database name 'lost+found'
2016-03-04 16:06:04 4033 [ERROR] Invalid (old?) table or database name 'lost+found'

[root@fuel ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/288647

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/288648

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/288648
Reason: medium importance bug

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/288647
Reason: medium importance bug

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/291916

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/291916
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0f92581240e338de996e2c5b7824ee08e26e23e3
Submitter: Jenkins
Branch: master

commit 0f92581240e338de996e2c5b7824ee08e26e23e3
Author: Maksim Malchuk <email address hidden>
Date: Sat Mar 12 00:02:06 2016 +0300

    Fix ignore-db-dir problem after switch from wsrep.cnf to my.cnf

    Switch from custom MySQL config to the global one in the Galera case
    would reproduce the old MySQL bug [1] when duplicate settings used
    when the mysql_install_db executed.

    This commit can be safely reverted when we switch puppetlabs-mysql
    to the head with PR#808 merged.

    [1] https://bugs.mysql.com/bug.php?id=69441

    Change-Id: Ie05c9b6dd664e6cf6fb891422de107b6d0f1c2e7
    Partial-Bug: #1484552

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/292507

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292507
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=521b733c94042eb2b8e795721b2bc42e0675dd7f
Submitter: Jenkins
Branch: master

commit 521b733c94042eb2b8e795721b2bc42e0675dd7f
Author: Maksim Malchuk <email address hidden>
Date: Mon Mar 14 19:51:29 2016 +0300

    Fix ignore-db-dir problem in puppetlabs-mysql

    This commit will swich to upstream puppetlabs-mysql module with fix.

    Change-Id: I2b13f471e420da7cea67759d44eda5ed217d50b8
    Partial-Bug: #1484552
    Closes-Bug: #1553955

[Alexander Rubtsov (arubtsov)]

We have customers who hit this bug at 7.0
Is it possible to produce the same fix for MOS 7.0 ?

[Alexander Rubtsov (arubtsov)]

sla2 for 7.0-updates

[Maksim Malchuk (mmalchuk)]

Alexander, this bug was originally fixed for 7.0 : https://review.openstack.org/#/c/231559/
how to reproduce it again? do you have an environment with the bug or diagnostic snapshot?

[Maksim Malchuk (mmalchuk)]

update: fix were reproduced and new fix proposed but not merged because of medium importance: https://review.openstack.org/#/c/288648/

[Vitaly Sedelnik (vsedelnik)]

Reopened and targeted to 7.0-mu-4 and 8.0-mu-2 per customer request.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288648
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6048308ff10f703df44c6d14dd59bea74a19690f
Submitter: Jenkins
Branch: stable/7.0

commit 6048308ff10f703df44c6d14dd59bea74a19690f
Author: Maksim Malchuk <email address hidden>
Date: Fri Mar 4 21:22:21 2016 +0300

    Add ignore_db_dirs support to Galera

    Previously we were setting ignore_db_dirs in the wsrep.cnf file for
    mysql. This is incorrect as this setting should be in the my.cnf.
    This change adds the ignore_db_dirs setting to the galera module
    which manages the my.cnf for the fuel galera deployment.

    Change-Id: I91399d8899b999622138f8408908ef0b2c5f1cf6
    Closes-Bug: #1484552
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit f302cb1ee1d55a24036b620ff01d57ca2e30014c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288647
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f302cb1ee1d55a24036b620ff01d57ca2e30014c
Submitter: Jenkins
Branch: stable/8.0

commit f302cb1ee1d55a24036b620ff01d57ca2e30014c
Author: Maksim Malchuk <email address hidden>
Date: Fri Mar 4 21:22:21 2016 +0300

    Add ignore_db_dirs support to Galera

    Previously we were setting ignore_db_dirs in the wsrep.cnf file for
    mysql. This is incorrect as this setting should be in the my.cnf.
    This change adds the ignore_db_dirs setting to the galera module
    which manages the my.cnf for the fuel galera deployment.

    Change-Id: I91399d8899b999622138f8408908ef0b2c5f1cf6
    Closes-Bug: #1484552
    Signed-off-by: Maksim Malchuk <email address hidden>

[Dmitry (dtsapikov)]

Verified on 7.0+mu4

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/329470

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/329822

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329470
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=a2f94378877fbeed33bbbc6d8d373ec809e570dd
Submitter: Jenkins
Branch: master

commit a2f94378877fbeed33bbbc6d8d373ec809e570dd
Author: Sergey Abramov <email address hidden>
Date: Tue Jun 14 17:11:37 2016 +0300

    Patch puppet

    * Add ignore_db_dirs config to puppet mysql manifest

    Change-Id: I6a777b2226aec07c823f3099f3c3b758fa056d2c
    Closes-bug: 1592373
    Related-bug: 1484552

[Maksym Strukov (unbelll)]

Verified as fixed in 9.0-mos-490

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329822
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=266b70f951c2dff55c88a28d6856218c53070ac5
Submitter: Jenkins
Branch: stable/8.0

commit 266b70f951c2dff55c88a28d6856218c53070ac5
Author: Sergey Abramov <email address hidden>
Date: Tue Jun 14 17:11:37 2016 +0300

    Patch puppet

    * Add ignore_db_dirs config to puppet mysql manifest

    Change-Id: I6a777b2226aec07c823f3099f3c3b758fa056d2c
    Closes-bug: 1592373
    Related-bug: 1484552
    (cherry picked from commit a2f94378877fbeed33bbbc6d8d373ec809e570dd)

[Ekaterina Shutova (eshutova)]

Verified on MOS 8.0 + mu2 updates.

Before updates:
mysql> show databases;
+---------------------+
| Database |
+---------------------+
| information_schema |
| cinder |
| glance |
| heat |
| keystone |
| #mysql50#lost+found |
| mysql |
| neutron |
| nova |
| performance_schema |
+---------------------+
10 rows in set (0.01 sec)

After updates:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| cinder |
| glance |
| heat |
| keystone |
| mysql |
| neutron |
| nova |
| performance_schema |
+--------------------+
9 rows in set (0.01 sec)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/332931

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332931
Committed: https://git.openstack.org/cgit/openstack/fuel-octane/commit/?id=24f3c396612eb0c69fbf06bab3cebbb9ca829426
Submitter: Jenkins
Branch: stable/mitaka

commit b34d749f1c281dafbdbf155bd86830dc8f2a9aa2
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 21:10:56 2016 +0300

    Support mock==1.8.0 in unit tests

    Use assert_called_once_with instead of assert_called_once that was
    introduced in 2.0.0.

    Change-Id: Ifb7699d4b552d148984961727355f0e23b487c7c

commit d60f1905143bb7576ffd670595de1c4aeafd7b34
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 02:01:52 2016 +0300

    Allow to authorize by the predefined admin_token

    The admin_token_auth middleware is added to keystone pipelines to
    allow authorization by admin_token.

    Change-Id: Ic03150305a669fad1446436a68051fb9aa25b892

commit cc7fab59f44ffef60285f8732d798f52469b2530
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:47:01 2016 +0300

    Reset default_domain_id before the keystone task

    The default_domain_id should be removed from keystone.conf after restore
    of DB and before to apply the keystone puppet task to avoid of using the
    configured domain as a default.

    Change-Id: I05a6c48532e8042496b3d8ccef53d65bf8c44653

commit e3f82399d567dbcfc1ae9a1ecbddba7bf5028fc8
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:32:47 2016 +0300

    Add helper function to iterate over parameters

    The helpers.iterate_parameters function allows to iterate over lines of
    INI-like files along with a context of information, such section,
    parameter and value.

    Change-Id: I55b179118116fd5dacf100754057ea6589782dc2

commit 5ed370a4fbb3369d0f75873d205ac7f0c655f93a
Author: Ilya Kharin <email address hidden>
Date: Wed Jun 22 01:18:05 2016 +0300

    Add update_file context manager for local files

    The subprocess.update_file function provides an ability to update
    content of a local file by iterating over lines of an original file and
    forming a result content in a temporary file to replace the original
    file in the end. This function is very useful to change configuration
    files.

    Change-Id: I433a5da67887b231400dd63131799019f45c277c

commit 58f31e6c5f408630c42565ee53de6b59457bc84c
Author: Oleg Gelbukh <email address hidden>
Date: Wed Jun 22 14:34:03 2016 +0000

    Escape passwords passed to openstack client

    If special symbols are used in password for 'admin' user in
    OpenStack, octane passes them to command line client as is
    and it breaks shell.

    Properly escape the password before passing it to subprocess.

    Change-Id: Iad635aec6d5b5cc32975937e00205b7e89dc99d9
    Closes-bug: 1585960

commit 896aba1191eeb59cf85cc8be6a2ae67e08b76070
Author: Yuriy Taraday <email address hidden>
Date: Wed Jun 22 15:28:45 2016 +0300

    Add absolute_import to util/docker.py to avoid local tempfile module

    Closes-Bug: 1595156
    Change-Id: I9484efce6fa7aec1b41cf592f9e9768d85931fa7

commit 46586a62df962b2ce00b5d8a63f6fd34c920a1f4
Author: Oleg Gelbukh <email address hidden>
Date: Thu M...