gcache.page. files not removed fast enough for some workloads

Bug #1488530 reported by Przemek on 2015-08-25
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Ivan Suzdal
6.1.x
High
Rodion Tikunov
7.0.x
High
Rodion Tikunov
8.0.x
High
Ivan Suzdal
Galera
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Confirmed
High
Unassigned

Bug Description

This is copy of initial report: https://github.com/codership/galera/issues/317

For workloads with huge transactions it may happen that gcache.page.xxxx files will fill all the available disk space as due to current removal algorithm, oldest files may remain pretty long.

Here is a quickly reproducible test case:

perl -e '$s="$s\x31"; for my $i (0..26214400) { print $s; } ' > /tmp/blob

use test
CREATE TABLE blob1 (
id int(11) NOT NULL,
a tinyint(4) DEFAULT NULL,
big longblob,
PRIMARY KEY (id)
) ENGINE=InnoDB;
insert into blob1 values (1,1,LOAD_FILE('/tmp/blob'));

for i in {1..50}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done

Changed in percona-xtradb-cluster:
milestone: none → 5.6.25-25.12
Przemek (pmalkowski) on 2015-09-08
Changed in percona-xtradb-cluster:
status: New → Confirmed
Changed in percona-xtradb-cluster:
milestone: 5.6.25-25.12 → future-5.6
Changed in percona-xtradb-cluster:
importance: Undecided → High
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → MOS Linux (mos-linux)
milestone: none → 9.0
Ivan Suzdal (isuzdal) on 2016-01-25
Changed in fuel:
assignee: MOS Linux (mos-linux) → Ivan Suzdal (isuzdal)

Related fix proposed to branch: master
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/16432

Reviewed: https://review.fuel-infra.org/16432
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: e7558b9c8ae2893ebef62cd5dd213b737c0b4f36
Author: Ivan Suzdal <email address hidden>
Date: Mon Jan 25 12:55:55 2016

Fix for gcache.page files removal.

Change-Id: Ia74799e9ea55a04a23cc0f13bab638522d1d08ec
Related-Bug: #1488530

Related fix proposed to branch: 8.0
Change author: Ivan Suzdal <email address hidden>
Review: https://review.fuel-infra.org/16499

Reviewed: https://review.fuel-infra.org/16499
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: 8acb72b37f8efa0b7a0911e4fd399dfcf8dadcd7
Author: Ivan Suzdal <email address hidden>
Date: Wed Jan 27 18:09:58 2016

Fix for gcache.page files removal.

Change-Id: Ia74799e9ea55a04a23cc0f13bab638522d1d08ec
Related-Bug: #1488530
(cherry picked from commit e7558b9c8ae2893ebef62cd5dd213b737c0b4f36)

Ivan Suzdal (isuzdal) on 2016-01-29
Changed in fuel:
status: Confirmed → Fix Committed

Verified om MOS 8.0 RC1:

mysql> use test
ERROR 1049 (42000): Unknown database 'test'
mysql> CREATE DATABASE test
    -> ;
Query OK, 1 row affected (0.00 sec)

mysql> use test
Database changed
mysql> CREATE TABLE blob1 (
    -> id int(11) NOT NULL,
    -> a tinyint(4) DEFAULT NULL,
    -> big longblob,
    -> PRIMARY KEY (id)
    -> ) ENGINE=InnoDB;
Query OK, 0 rows affected (0.03 sec)

mysql> insert into blob1 values (1,1,LOAD_FILE('/tmp/blob'));
ERROR 1118 (42000): The size of BLOB/TEXT data inserted in one transaction is greater than 10% of redo log size. Increase the redo log size using innodb_log_file_size.

So, the size of this file was reduced.

issue not reproduced after
root@node-1:~# for i in {1..50}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done
root@node-1:~# for i in {1..5000}; do mysql test -e "UPDATE blob1 SET a=$i,big=LOAD_FILE('/tmp/blob')"; done
root@node-1:~#

Moved to Fix Released.

Rodion Tikunov (rtikunov) wrote :

In 7.0 galera updated to galera-3 and the bug can not be reproduced.

Rodion Tikunov (rtikunov) wrote :

Could not reproduce for 6.1

Alexander Gubanov (ogubanov) wrote :

Verified on MOS 9.0 (build 201) - can't reproduce.
Details: http://pastebin.com/fiKp86nA

Changed in fuel:
status: Fix Committed → Fix Released
Claudio (claudio-enjoy) wrote :

same issue with :

Percona-XtraDB-Cluster-client-56-5.6.35-26.20.2.el6.x86_64
Percona-XtraDB-Cluster-shared-56-5.6.35-26.20.2.el6.x86_64
Percona-XtraDB-Cluster-56-5.6.35-26.20.2.el6.x86_64
percona-toolkit-2.2.15-2.noarch
Percona-XtraDB-Cluster-galera-3-3.20-2.el6.x86_64
percona-xtrabackup-2.3.5-1.el6.x86_64
Percona-XtraDB-Cluster-server-56-5.6.35-26.20.2.el6.x86_64

both configuration generate 300+ gcache.page.000* logs

1 wsrep_provider_options = gcache.size=2G;gcache.keep_pages_size=100M;gcache.keep_pages_count=20
2 wsrep_provider_options = gcache.size=2G;gcache.keep_pages_size=2GB;gcache.keep_pages_count=10

any suggestion ?

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers