| 2015-07-26 07:06:29 |
Sam Stoelinga |
bug |
|
|
added bug |
| 2015-07-26 07:08:42 |
Sam Stoelinga |
description |
Description: If one of the nodes fail in the MySQL galera cluster there is sometimes the need for full state transfer which takes more than 10 minutes if the database is large e.g. when using Zabbix.
Current situation:
The mysql database is unable to start because it gets killed before full state transfer is finished.
Expected situation:
Instead of increasing the timeout we should do a special check to see whether a state transfer is still in progress if so notify pacemaker to not yet kill the mysql starting process. IF this is not possible we can consider changing timeout from 600 seconds to higher, but i suggest instead we decrease the timeout and use a specific check to see if SST is in process.
Steps to reproduce:
1. Create an HA environment
2. Import 10+Gb of data into the mysql database
3. Let one of the nodes fail on purpose and require full state transfer. Detailed steps for this:
kill mysqld_safe and mysqld process && rm /var/lib/mysql/* -rf
4. Wait for corosync/pacemaker to restart the mysql process on specified node.
5. The log on failed node should show: WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr,nodelay,sndbuf=1048576,rcvbuf=1048576 stdio |
Description: If one of the nodes fail in the MySQL galera cluster there is sometimes the need for full state transfer which takes more than 10 minutes if the database is large e.g. when using Zabbix.
Current situation:
The mysql database is unable to start because it gets killed before full state transfer is finished.
Expected situation:
Instead of increasing the timeout we should do a special check to see whether a state transfer is still in progress if so notify pacemaker to not yet kill the mysql starting process. IF this is not possible we can consider changing timeout from 600 seconds to higher, but i suggest instead we decrease the timeout and use a specific check to see if SST is in process.
Steps to reproduce:
1. Create an HA environment
2. Import 10+Gb of data into the mysql database
3. Let one of the nodes fail on purpose and require full state transfer. Detailed steps for this:
kill mysqld_safe and mysqld process && rm /var/lib/mysql/* -rf
4. Wait for corosync/pacemaker to restart the mysql process on specified node.
5. The log on failed node should show: WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr,nodelay,sndbuf=1048576,rcvbuf=1048576 stdio
Current workaround:
crm configure edit p_mysql
Chanage "op start interval=0 timeout=600"
to "op start interval=0 timeout=1600" |
|
| 2015-07-27 05:47:38 |
Nastya Urlapova |
fuel: milestone |
|
7.0 |
|
| 2015-07-27 05:47:51 |
Nastya Urlapova |
fuel: assignee |
|
Fuel Library Team (fuel-library) |
|
| 2015-07-27 05:47:55 |
Nastya Urlapova |
fuel: status |
New |
Confirmed |
|
| 2015-07-27 05:47:58 |
Nastya Urlapova |
fuel: importance |
Undecided |
High |
|
| 2015-07-27 14:53:49 |
Oleksiy Molchanov |
fuel: assignee |
Fuel Library Team (fuel-library) |
Sergii Golovatiuk (sgolovatiuk) |
|
| 2015-08-03 15:38:33 |
Sergii Golovatiuk |
fuel: status |
Confirmed |
In Progress |
|
| 2015-08-04 14:32:32 |
ruhe |
fuel: assignee |
Sergii Golovatiuk (sgolovatiuk) |
Denis Egorenko (degorenko) |
|
| 2015-08-04 20:38:56 |
Sergii Golovatiuk |
fuel: assignee |
Denis Egorenko (degorenko) |
Sergii Golovatiuk (sgolovatiuk) |
|
| 2015-08-05 10:25:11 |
Adam Heczko |
bug |
|
|
added subscriber Adam Heczko |
| 2015-08-14 11:31:09 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2015-09-14 11:56:53 |
Oleksiy Molchanov |
tags |
|
on-verification |
|
| 2015-09-14 13:47:18 |
Oleksiy Molchanov |
tags |
on-verification |
|
|
| 2015-09-14 13:47:21 |
Oleksiy Molchanov |
fuel: status |
Fix Committed |
Fix Released |
|
| 2016-01-18 21:21:44 |
Andrii Petrenko |
tags |
|
customer-found support |
|
| 2016-01-20 21:10:04 |
Sergii Golovatiuk |
fuel: status |
Fix Released |
Confirmed |
|
| 2016-01-20 21:10:17 |
Sergii Golovatiuk |
nominated for series |
|
fuel/8.0.x |
|
| 2016-01-20 21:10:17 |
Sergii Golovatiuk |
bug task added |
|
fuel/8.0.x |
|
| 2016-01-20 21:10:17 |
Sergii Golovatiuk |
nominated for series |
|
fuel/mitaka |
|
| 2016-01-20 21:10:17 |
Sergii Golovatiuk |
bug task added |
|
fuel/mitaka |
|
| 2016-01-20 21:10:27 |
Sergii Golovatiuk |
fuel/8.0.x: status |
New |
Confirmed |
|
| 2016-01-20 21:10:33 |
Sergii Golovatiuk |
nominated for series |
|
fuel/7.0.x |
|
| 2016-01-20 21:10:33 |
Sergii Golovatiuk |
bug task added |
|
fuel/7.0.x |
|
| 2016-01-20 21:10:39 |
Sergii Golovatiuk |
fuel/7.0.x: status |
New |
Confirmed |
|
| 2016-01-20 21:10:42 |
Sergii Golovatiuk |
fuel/8.0.x: importance |
Undecided |
High |
|
| 2016-01-20 21:10:43 |
Sergii Golovatiuk |
fuel/7.0.x: importance |
Undecided |
High |
|
| 2016-01-20 21:10:45 |
Sergii Golovatiuk |
fuel/8.0.x: assignee |
|
Sergii Golovatiuk (sgolovatiuk) |
|
| 2016-01-20 21:10:47 |
Sergii Golovatiuk |
fuel/7.0.x: assignee |
|
Sergii Golovatiuk (sgolovatiuk) |
|
| 2016-01-20 21:10:52 |
Sergii Golovatiuk |
fuel/8.0.x: milestone |
|
8.0 |
|
| 2016-01-20 21:10:54 |
Sergii Golovatiuk |
fuel/7.0.x: milestone |
|
9.0 |
|
| 2016-01-21 08:40:02 |
Richard Berwald |
bug |
|
|
added subscriber Richard Berwald |
| 2016-01-21 18:00:09 |
Dmitry Pyzhov |
bug task deleted |
fuel/mitaka |
|
|
| 2016-01-21 18:00:19 |
Dmitry Pyzhov |
fuel: milestone |
7.0 |
9.0 |
|
| 2016-01-21 18:00:24 |
Dmitry Pyzhov |
fuel/7.0.x: milestone |
9.0 |
7.0-updates |
|
| 2016-01-21 18:00:34 |
Dmitry Pyzhov |
tags |
customer-found support |
area-library customer-found support |
|
| 2016-01-26 10:54:02 |
Matthew Mosesohn |
tags |
area-library customer-found support |
area-library customer-found support team-bugfix |
|
| 2016-01-26 17:31:45 |
OpenStack Infra |
fuel: status |
Confirmed |
In Progress |
|
| 2016-01-27 16:10:10 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2016-01-28 10:04:20 |
Matthew Mosesohn |
fuel/8.0.x: status |
Confirmed |
In Progress |
|
| 2016-01-28 11:03:27 |
Sergii Golovatiuk |
fuel/8.0.x: status |
In Progress |
Fix Committed |
|
| 2016-01-28 22:01:06 |
Evgeniya Shumakher |
tags |
area-library customer-found support team-bugfix |
area-docs area-library customer-found release-notes support team-bugfix |
|
| 2016-02-23 04:50:19 |
Andrii Petrenko |
nominated for series |
|
fuel/6.1.x |
|
| 2016-02-23 04:50:19 |
Andrii Petrenko |
bug task added |
|
fuel/6.1.x |
|
| 2016-02-23 04:54:27 |
Andrii Petrenko |
fuel/6.1.x: importance |
Undecided |
High |
|
| 2016-02-23 04:54:34 |
Andrii Petrenko |
fuel/6.1.x: milestone |
|
6.1-updates |
|
| 2016-02-24 08:46:25 |
Dmitry Klenov |
fuel/6.1.x: assignee |
|
MOS Maintenance (mos-maintenance) |
|
| 2016-02-24 08:46:29 |
Dmitry Klenov |
fuel/6.1.x: status |
New |
Confirmed |
|
| 2016-02-24 15:28:37 |
Rodion Tikunov |
fuel/6.1.x: assignee |
MOS Maintenance (mos-maintenance) |
Rodion Tikunov (rtikunov) |
|
| 2016-02-25 09:25:09 |
Rodion Tikunov |
fuel/6.1.x: status |
Confirmed |
In Progress |
|
| 2016-02-26 08:57:30 |
Rodion Tikunov |
fuel/7.0.x: assignee |
Sergii Golovatiuk (sgolovatiuk) |
Rodion Tikunov (rtikunov) |
|
| 2016-02-26 08:59:38 |
Rodion Tikunov |
fuel/7.0.x: status |
Confirmed |
In Progress |
|
| 2016-02-26 09:29:05 |
Rodion Tikunov |
fuel/7.0.x: milestone |
7.0-updates |
7.0-mu-3 |
|
| 2016-03-04 09:25:55 |
Rodion Tikunov |
fuel/6.1.x: milestone |
6.1-updates |
6.1-mu-6 |
|
| 2016-03-15 17:13:31 |
Denis Meltsaykin |
fuel/7.0.x: status |
In Progress |
Fix Committed |
|
| 2016-03-30 06:00:57 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found release-notes support team-bugfix |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
|
| 2016-03-30 17:58:59 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
area-docs area-library customer-found release-notes support team-bugfix |
|
| 2016-03-30 17:59:18 |
TatyanaGladysheva |
fuel/7.0.x: status |
Fix Committed |
Fix Released |
|
| 2016-04-25 11:57:28 |
Rodion Tikunov |
fuel/6.1.x: status |
In Progress |
Fix Committed |
|
| 2016-05-12 10:07:40 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found release-notes support team-bugfix |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
|
| 2016-05-17 12:55:48 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
area-docs area-library customer-found release-notes support team-bugfix |
|
| 2016-05-17 12:56:05 |
TatyanaGladysheva |
fuel/6.1.x: status |
Fix Committed |
Fix Released |
|
| 2016-06-08 11:31:51 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found release-notes support team-bugfix |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
|
| 2016-06-08 17:03:14 |
TatyanaGladysheva |
tags |
area-docs area-library customer-found on-verification release-notes support team-bugfix |
area-docs area-library customer-found release-notes support team-bugfix |
|
| 2016-06-08 17:03:30 |
TatyanaGladysheva |
fuel: status |
Fix Committed |
Fix Released |
|
| 2016-06-30 11:52:20 |
Evgeny Konstantinov |
tags |
area-docs area-library customer-found release-notes support team-bugfix |
area-docs area-library customer-found release-notes-done support team-bugfix |
|
