FIFO size exceeds physical memory limit

Bug #1181347 reported by Yannick Robin
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Galera
Fix Released
Medium
Alex Yurchenko

Bug Description

CONFIGURATION
****************

Ubuntu 10.2 32-bit
Percona 5.5 xtra DB Cluster

ISSUE
*****

The node fails to restart because of FIFO size exceeds physical memory limit error.

Here are the logs :

130517 06:33:39 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
130517 06:33:39 mysqld_safe WSREP: Running position recovery with --log_error=/tmp/tmp.nmPV61saEx
130517 06:33:45 mysqld_safe WSREP: Recovered position 36033c88-be90-11e2-0800-df7539603f95:347
130517 6:33:45 [Note] WSREP: wsrep_start_position var submitted: '36033c88-be90-11e2-0800-df7539603f95:347'
130517 6:33:45 [Note] WSREP: Read nil XID from storage engines, skipping position init
130517 6:33:45 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so'
130517 6:33:45 [Note] WSREP: wsrep_load(): Galera 2.5(r150) by Codership Oy <email address hidden> loaded succesfully.
130517 6:33:45 [Note] WSREP: Found saved state: 36033c88-be90-11e2-0800-df7539603f95:347
130517 6:33:45 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130517 6:33:45 [ERROR] WSREP: galerautils/src/gu_fifo.c:gu_fifo_create():94: Resulting FIFO size 1208025276 exceeds physical memory limit 1001381888
130517 6:33:45 [ERROR] WSREP: gcs/src/gcs.c:gcs_create():265: Failed to create recv_q.
130517 6:33:45 [ERROR] WSREP: gcs/src/gcs.c:gcs_create():311: Failed to create GCS connection handle.
130517 6:33:45 [Note] WSREP: Passing config to GCS: base_host = 192.168.0.11; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130517 6:33:45 [Note] WSREP: Flushing memory map to disk...
130517 6:33:46 [ERROR] WSREP: could not create gcs connection (FATAL)
         at galera/src/gcs.hpp:Gcs():73
130517 6:33:46 [ERROR] WSREP: wsrep::init() failed: 6, must shutdown
130517 6:33:46 [ERROR] Aborting

130517 6:33:46 [Note] /usr/sbin/mysqld: Shutdown complete

130517 06:33:46 mysqld_safe mysqld from pid file /var/lib/mysql/ubuntu.pid ended

SOLUTION
*********

The physical memory indicated (1001381888) is wrong. I have 5GB. Here is the result of getconf :

ubuntu@ubuntu:/home/hybris/hybris/config$ getconf -a | grep PAGE
PAGESIZE 4096
PAGE_SIZE 4096
_AVPHYS_PAGES 778690
_PHYS_PAGES 1293054

See gu_fifo.c code :
   mem_limit = GU_PHYS_PAGES * GU_PAGE_SIZE;

The value is wrong because an arithmetic overfow for 32-bit system :
5296349184 (max memory of my system) - 4294967295 (maximum representable value of 32bits number limit) = 1001381889 (physical limit indicated by Galera)

Thanks,

Yannick Robin

Changed in galera:
assignee: nobody → Alex Yurchenko (ayurchen)
importance: Undecided → Medium
status: New → In Progress
Revision history for this message
stefanita rares dumitrescu (katmai-d) wrote :

hi,

i am experiencing the same issue on a similar server. is there any way to fix it or at least get the sql back online?

regards

Revision history for this message
Yannick Robin (yannick-poubelle) wrote :

The workaround is to reduce the physical memory of the server (<4GB).

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Fix committed in r152

Changed in galera:
status: In Progress → Fix Committed
milestone: none → 23.2.6
Changed in galera:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.