Using too big key for internal temp tables & Out Of Memory Error in MariaDB 5.5

[DaveFM]

Executing this query on MariaDB errors out - and occasionally works - MySQL 5.5, and MariaDB 5.3.5 handle this query perfectly.
Issue happening with 5.5.21 Beta.

I've tried changing various memory related settings in my.cnf without any success.. If needed I can upload the tables related to this query.

MariaDB [sabretooth]> SELECT channels.id, channels.created, channels.vhost, channels.name, channels.shortname, channels.parentchannel, channels.thumburl, channels.status, channels.description, channels.readonly, channels.disableembed, channels.modemailaccepted, channels.modemaildenied, channels.emailaddress, channels.defaultsort, CASE channels.visibility WHEN 1 THEN 'SHOWN' WHEN 0 THEN 'HIDDEN' END AS visibility, geoblocking.countrycodes AS geoblocking FROM channels LEFT JOIN ( SELECT GROUP_CONCAT( countrycode ) AS countrycodes, channelid FROM channels_geoblocking GROUP BY channelid ) AS geoblocking ON channels.id = geoblocking.channelid WHERE channels.vhost = 227 AND channels.status != 7 AND ( 1 = 1 ) ORDER BY channels.parentchannel, channels.name, channels.id;
ERROR 5 (HY000): Out of memory (Needed 4212250480 bytes)

MariaDB [sabretooth]> SELECT channels.id, channels.created, channels.vhost, channels.name, channels.shortname, channels.parentchannel, channels.thumburl, channels.status, channels.description, channels.readonly, channels.disableembed, channels.modemailaccepted, channels.modemaildenied, channels.emailaddress, channels.defaultsort, CASE channels.visibility WHEN 1 THEN 'SHOWN' WHEN 0 THEN 'HIDDEN' END AS visibility, geoblocking.countrycodes AS geoblocking FROM channels LEFT JOIN ( SELECT GROUP_CONCAT( countrycode ) AS countrycodes, channelid FROM channels_geoblocking GROUP BY channelid ) AS geoblocking ON channels.id = geoblocking.channelid WHERE channels.vhost = 227 AND channels.status != 7 AND ( 1 = 1 ) ORDER BY channels.parentchannel, channels.name, channels.id;
ERROR 5 (HY000): Out of memory (Needed 3527552248 bytes)

MariaDB [sabretooth]> SELECT channels.id, channels.created, channels.vhost, channels.name, channels.shortname, channels.parentchannel, channels.thumburl, channels.status, channels.description, channels.readonly, channels.disableembed, channels.modemailaccepted, channels.modemaildenied, channels.emailaddress, channels.defaultsort, CASE channels.visibility WHEN 1 THEN 'SHOWN' WHEN 0 THEN 'HIDDEN' END AS visibility, geoblocking.countrycodes AS geoblocking FROM channels LEFT JOIN ( SELECT GROUP_CONCAT( countrycode ) AS countrycodes, channelid FROM channels_geoblocking GROUP BY channelid ) AS geoblocking ON channels.id = geoblocking.channelid WHERE channels.vhost = 227 AND channels.status != 7 AND ( 1 = 1 ) ORDER BY channels.parentchannel, channels.name, channels.id;
^[[A
ERROR 1928 (HY000): Internal error: 'Using too big key for internal temp tables'

MariaDB [sabretooth]> SELECT channels.id, channels.created, channels.vhost, channels.name, channels.shortname, channels.parentchannel, channels.thumburl, channels.status, channels.description, channels.readonly, channels.disableembed, channels.modemailaccepted, channels.modemaildenied, channels.emailaddress, channels.defaultsort, CASE channels.visibility WHEN 1 THEN 'SHOWN' WHEN 0 THEN 'HIDDEN' END AS visibility, geoblocking.countrycodes AS geoblocking FROM channels LEFT JOIN ( SELECT GROUP_CONCAT( countrycode ) AS countrycodes, channelid FROM channels_geoblocking GROUP BY channelid ) AS geoblocking ON channels.id = geoblocking.channelid WHERE channels.vhost = 227 AND channels.status != 7 AND ( 1 = 1 ) ORDER BY channels.parentchannel, channels.name, channels.id;
ERROR 5 (HY000): Out of memory (Needed 3527552248 bytes)

My.cnf
---------
#
# The MySQL database server configuration file.
#
# You can copy this to one of:
# - "/etc/mysql/my.cnf" to set global options,
# - "/var/lib/mysql/my.cnf" to set server-specific options or
# - "~/.my.cnf" to set user-specific options.
#
# One can use all long options that the program supports.
# Run program with --help to get a list of available options and with
# --print-defaults to see which it would actually understand and use.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

# This will be passed to all mysql clients
# It has been reported that passwords should be enclosed with ticks/quotes
# escpecially if they contain "#" chars...
# Remember to edit /etc/mysql/debian.cnf when changing the socket location.
[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

# Here is entries for some specific programs
# The following values assume you have at least 32M ram

# This was formally known as [safe_mysqld]. Both versions are currently parsed.
[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0
max_connections = 8000

[mysqld]
#
# * Basic Settings
#
user = mysql
thread_stack = 2M
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql/
tmpdir = /tmp
language = /usr/share/mysql/english
skip-external-locking
skip-name-resolve
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1
bind-address = 0.0.0.0
#
# * Fine Tuning
#
key_buffer = 64M
max_allowed_packet = 16M
#
# * Query Cache Configuration
#
query_cache_limit = 1048576
query_cache_size = 67108864
query_cache_type = 1
#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
#log = /var/log/mysql.log
#log = /var/log/mysql/mysql.log
#
# Error logging goes to syslog. This is a Debian improvement
#
# Here you can see queries with especially long duration
log-slow-queries = /var/log/mysql/mysql-slow.log
long_query_time = 1
#
# The following can be used as easy to replay backup logs or for replication.
server-id = 23
log-bin = /var/log/mysql/mysql-bin.log
# See /etc/mysql/debian-log-rotate.conf for the number of files kept.
max_binlog_size = 104857600
#binlog-do-db = include_database_name
#binlog-ignore-db = include_database_name
#
# * BerkeleyDB
#
# The use of BerkeleyDB is now discouraged and support for it will probably
# cease in the next versions.
#
# * InnoDB
#
# InnoDB is enabled by default with a 10MB datafile in /var/lib/mysql/.
# Read the manual for more InnoDB related options. There are many!
#
# * Security Feature
#
# Read the manual, too, if you want chroot!
# chroot = /var/lib/mysql/
#
# If you want to enable SSL support (recommended) read the manual or my
# HOWTO in /usr/share/doc/mysql-server/SSL-MINI-HOWTO.txt.gz
# ssl-ca=/etc/mysql/cacert.pem
# ssl-cert=/etc/mysql/server-cert.pem
# ssl-key=/etc/mysql/server-key.pem
max-connections = 8000
innodb_buffer_pool_size = 1G
#innodb_log_file_size=250M
#innodb_log_buffer_size=8M

[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

[DaveFM (dave-minogue)]

I've uploaded the tables needed for this to private/958033.tgz

[Elena Stepanova (elenst)]

Hi David,

Thanks for the test case.

Please try to set optimizer_switch=derived_with_keys=off, hopefully it will help with the problem for now; I tried your test case, the failure was reproducible with the default optimizer_switch, but not with derived_with_keys=off.

Although with your variation the problem only reveals on 5.5, I am fairly certain it is the same problem as described in bug https://bugs.launchpad.net/maria/+bug/957409 and which was fixed along with bugfix https://bugs.launchpad.net/maria/+bug/953649:
- it produces the same or very similar valgrind warnings as described in 957409;
- derived_with_keys=off seems to make it go away;
- after the latest merge 5.3->5.5, which includes bugfix https://bugs.launchpad.net/maria/+bug/953649, it is not reproducible any longer.

If you confirm that derived_with_keys=off helped, I will close this bug as a duplicate of https://bugs.launchpad.net/maria/+bug/957409.

Thank you.

[Elena Stepanova (elenst)]

As discussed on IRC, setting derived_with_keys=off helped indeed.
So, this bug is another representation of bug #957409 and was fixed with bug #953649, the fix is now both in maria/5.3 and maria/5.5.