Bad permissions on DATADIR/.temporary causes Segmentation Fault

[BJ Dierkes]

I ran into an obscure segmentation fault after [most likely] inadvertently running drizzled as root instead of as 'drizzle':

[root@f12-drizzle ~]# sudo -u drizzle drizzled
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
100623 23:15:29 - drizzled got signal 11;
This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

GDB Back Trace (with some debug symbols missing):

-bash-4.0$ gdb drizzled
GNU gdb (GDB) Fedora (7.0.1-48.fc12)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/drizzled...Reading symbols from /usr/lib/debug/usr/sbin/drizzled.debug...done.
done.

(gdb) run
Starting program: /usr/sbin/drizzled
[Thread debugging using libthread_db enabled]
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
Program received signal SIGSEGV, Segmentation fault.

(gdb) bt full
#0 0x0000000000000000 in ?? ()
No symbol table info available.
#1 0x00000000006722ab in drizzled::my_error (nr=2, MyFlags=36) at drizzled/error.cc:140
        format = "Can't create file '%-.200s' (errno: %d)"
        args = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffffffb1c0,
            reg_save_area = 0x7fffffffb0d0}}
        ebuff = "Can't create file '/var/lib/drizzle//.temporary/ibdDJXTn' (errno: 13)\000\000\000\350\340\377\367\377\177\000\000 \336\376\367\377\177\000\000K\214\336\367\377\177\000\000\000\000\000\000\000\000\000\000 \336\376\367\377\177\000\000\001", '\000' <repeats 15 times>, "\001\000\000\000\000\000\000\000\350\340\377\367\377\177\000\000\b\000\000\000\000\000\000\000@\253\264\000\000\000\000\000\000\335\377\377\377\177\000\000\020ߺ\000\000\000\000\000\260\341\377\377\377\177\000\000\000\000\000\000\000\000\000\000@\344\377\367\377\177\000\000\360\260\377\377\377\177\000\000\b\261\377\377\377\177\000\000\000\000\000\000\001\000\000\000\346\320E\314\000\000\000\000\244\030F", '\000' <repeats 21 times>, "o\262\377\377\377\177\000\000*\n\332\366\377\177\000\000[\000\000"
#2 0x00000000006554f4 in drizzled::internal::my_register_filename (fd=<value optimized out>,
    FileName=0x24 <Address 0x24 out of bounds>, error_message_number=<value optimized out>,
    MyFlags=<value optimized out>) at drizzled/internal/my_open.cc:124
No locals.
#3 0x0000000000654d6e in drizzled::internal::create_temp_file (
    to=0x7fffffffb250 "/var/lib/drizzle//.temporary/ibdDJXTn",
    dir=0xbae158 "/var/lib/drizzle//.temporary", prefix=<value optimized out>, MyFlags=16)
    at drizzled/internal/mf_tempfile.cc:86
        file = -1
        org_file = -1
        prefix_str = "ibXXXXXX"
#4 0x00000000005c7965 in drizzled::mysql_tmpfile (prefix=0x24 <Address 0x24 out of bounds>)
    at drizzled/session.cc:111
        filename = "/var/lib/drizzle//.temporary/ibdDJXTn\000\000\000\000\320\376\367\377\177\000\000\367\031F\000\000\000\000\000`\266\321\366\377\177\000\000\350\177A\000\000\000\000\000\000\000\000\000\001\000\000\000\311\003\000\000\001\000\000\000\002", '\000' <repeats 15 times>"\240, ", '\000' <repeats 23 times>, "p\263\377\377\377\177\000\000\b\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000P\002", '\000' <repeats 14 times>, "%\032\336\366\377\177\000\000\200\376\a\367\377\177\000\000\200\000\000\000\000\000\000\000p\017\376\000\000\000\000\000\331h\330\366\377\177\000\000\b\000\000\000\000\000\000\000c7\330\366\377\177\000\000X\270\352\000\000\000\000\000x\000\000\000\000\000\000\000\260\263\377\377\377\177\000\000\002", '\000' <repeats 15 times>"\240, \000\000\000\000\000\000\000\000\264"...
        fd = <value optimized out>
#5 0x00000000006a3547 in innobase_mysql_tmpfile () at plugin/innobase/handler/ha_innodb.cc:1406

THE FIX:

Setting the ownership on /var/lib/drizzle/.temporary to the 'drizzle' user resolved the issue:

[root@f12-drizzle ~]# chown -R drizzle:drizzle /var/lib/drizzle/.temporary/

[root@f12-drizzle ~]# sudo -u drizzle drizzled
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
100623 23:14:45 InnoDB: highest supported file format is Barracuda.
100623 23:14:45 InnoDB Plugin 1.0.4 started; log sequence number 44534
Listening on 0.0.0.0:9306
Listening on :::9306
Listening on 0.0.0.0:3306
Listening on :::3306
drizzled: Forcing close of thread 0 user: ''
drizzled: ready for connections.
Version: '2010.06.1634' Source distribution (trunk)
drizzled: Normal shutdown
drizzled: Shutdown complete
100623 23:15:00 InnoDB: Starting shutdown...
100623 23:15:00 InnoDB: Shutdown completed; log sequence number 44560

SUGGESTION:

Properly check for the ability to read/write on DATADIR/.temporary and catch the error condition.. providing a useful error message to the user (rather than allowing a segmentation fault condition)

[BJ Dierkes (derks)]

Actually... I see that this was caught by drizzled/error.cc .... Any reason this is not displayed to the user?

[BJ Dierkes (derks)]

I actually think there might be another issue here that I'm not sure should be resolved in drizzle, or via package maintainers.

a) Starting up drizzled for the first time creates the database [of missing] in DATADIR

b) If run as root, even with 'user=drizzle' in drizzled.cnf ... the DATADIR/.temporary directory is created by root before switching the context to the 'drizzle' user (as per the config).

c) drizzled seg faults because DATADIR/.temporary is owned by root... but drizzled is running as the 'drizzle' user at this point.

Basically, what this means is... that package maintainers would need to 'chown -R $DATADIR' on startup to ensure the files in DATADIR are owned by the drizzle owner... something that has been a big stink for mysql users on Redhat/Fedora in the past.

SUGGESTION:

The process should be handed over to the 'user' as per the config before any files in DATADIR are created (specifically .temporary).

[Brian Aker (brianaker)]

Hi!

On Jun 23, 2010, at 4:44 PM, BJ Dierkes wrote:

> The process should be handed over to the 'user' as per the config before
> any files in DATADIR are created (specifically .temporary).

So you are suggesting that we do the test/create after we have switched users?

Cheers,
 -Brian

[Stewart Smith (stewart)]

On Thu, 24 Jun 2010 02:08:13 -0000, Brian Aker <email address hidden> wrote:
> Hi!
>
> On Jun 23, 2010, at 4:44 PM, BJ Dierkes wrote:
>
> > The process should be handed over to the 'user' as per the config before
> > any files in DATADIR are created (specifically .temporary).
>
> So you are suggesting that we do the test/create after we have switched
> users?

I'll also suggest we don't segfault and report a decent error message :)

--
Stewart Smith

[BJ Dierkes (derks)]

On Jun 23, 2010, at 9:08 PM, Brian Aker wrote:
>
> So you are suggesting that we do the test/create after we have switched
> users?
>

More or less, yes. drizzled runs as the user specified in drizzed.cnf... I.e.:

[drizzled]
user = drizzle

Meaning, under this scenario, if I run 'drizzled' as 'root' I can assume that the process will start as root, then switch to the drizzle user, then run... but right now it bails because the initial .temporary is created as root. The obvious answer is 'just start the process as the drizzle user' but this doesn't really sound ideal to me being that all the pieces are already there... just that some of them are out of order.

Thanks.

---
derks

[Andrew Hutchings (linuxjedi)]

Not sure how to express this in a dtr. My best guess would be to create a .temporary directory, chown 000 and start drizzled which would cover part of this. But how do you create a directory and chown before drizzled is started? (I am guessing the dtr test case has to shut down drizzled and start it again?).

[Patrick Crews (patrick-crews)]

I would say that you are on the right track. The test language has
shutdown_server [timeout], but we don't have a handy start_server. The
MySQL test suite has some examples of this kind of stuff in the rpl suite
+ t/crash_commit_before.test. Hope that helps. I'll be around more
tomorrow if you need any help.

On Sat, Jul 10, 2010 at 4:43 AM, LinuxJedi <email address hidden> wrote:

> Not sure how to express this in a dtr. My best guess would be to create
> a .temporary directory, chown 000 and start drizzled which would cover
> part of this. But how do you create a directory and chown before
> drizzled is started? (I am guessing the dtr test case has to shut down
> drizzled and start it again?).
>
> --
> Bad permissions on DATADIR/.temporary causes Segmentation Fault
> https://bugs.launchpad.net/bugs/597905
> You received this bug notification because you are a member of Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Fix Committed
>
> Bug description:
> I ran into an obscure segmentation fault after [most likely] inadvertently
> running drizzled as root instead of as 'drizzle':
>
> [root@f12-drizzle ~]# sudo -u drizzle drizzled
> InnoDB: The InnoDB memory heap is disabled
> InnoDB: Mutexes and rw_locks use GCC atomic builtins.
> 100623 23:15:29 - drizzled got signal 11;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly
> built,
> or misconfigured. This error can also be caused by malfunctioning
> hardware.
> We will try our best to scrape up some info that will hopefully help
> diagnose
> the problem, but since we have already crashed, something is definitely
> wrong
> and this may fail.
>
> GDB Back Trace (with some debug symbols missing):
>
> -bash-4.0$ gdb drizzled
> GNU gdb (GDB) Fedora (7.0.1-48.fc12)
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/drizzled...Reading symbols from
> /usr/lib/debug/usr/sbin/drizzled.debug...done.
> done.
>
> (gdb) run
> Starting program: /usr/sbin/drizzled
> [Thread debugging using libthread_db enabled]
> InnoDB: The InnoDB memory heap is disabled
> InnoDB: Mutexes and rw_locks use GCC atomic builtins.
> Program received signal SIGSEGV, Segmentation fault.
>
> (gdb) bt full
> #0 0x0000000000000000 in ?? ()
> No symbol table info available.
> #1 0x00000000006722ab in drizzled::my_error (nr=2, MyFlags=36) at
> drizzled/error.cc:140
> format = "Can't create file '%-.200s' (errno: %d)"
> args = {{gp_offset = 32, fp_offset = 48, overflow_arg_area =
> 0x7fffffffb1c0,
> reg_save_area = 0x7fffffffb0d0}}
> ebuff = "Can't create file '/var/lib/drizzle//.temporary/ibdDJXTn'
> (...

[Andrew Hutchings (linuxjedi)]

Hi Patrick,

I did something similar for a cluster mtr test case not too long ago. I'll try it with dtr later today and see where we get.

[Monty Taylor (mordred)]

On 07/10/2010 06:06 AM, LinuxJedi wrote:
> Hi Patrick,
>
> I did something similar for a cluster mtr test case not too long ago.
> I'll try it with dtr later today and see where we get.

If you figure this out (doing something before server start) then we'll
be able to properly express the server startup tests jay was wanting in
his replication branch, as well as being able to write tests for server
options.

[Andrew Hutchings (linuxjedi)]

We currently have the option for test_name.sh files which are run before drizzled is executed. This could help in some cases but I should think it will not be too hard to add some kind of override script type with a new extension. Maybe a worklog for someone?

[Andrew Hutchings (linuxjedi)]

Doh! I mean test_name-master.sh!

[Brian Aker (brianaker)]

You can always start up an additional server to test this during startup.

On Jul 10, 2010, at 1:36 PM, LinuxJedi wrote:

> We currently have the option for test_name.sh files which are run before
> drizzled is executed. This could help in some cases but I should think
> it will not be too hard to add some kind of override script type with a
> new extension. Maybe a worklog for someone?
>
> --
> Bad permissions on DATADIR/.temporary causes Segmentation Fault
> https://bugs.launchpad.net/bugs/597905
> You received this bug notification because you are a member of Drizzle-
> developers, which is subscribed to Drizzle.
>
> Status in A Lightweight SQL Database for Cloud and Web: Fix Committed
>
> Bug description:
> I ran into an obscure segmentation fault after [most likely] inadvertently running drizzled as root instead of as 'drizzle':
>
> [root@f12-drizzle ~]# sudo -u drizzle drizzled
> InnoDB: The InnoDB memory heap is disabled
> InnoDB: Mutexes and rw_locks use GCC atomic builtins.
> 100623 23:15:29 - drizzled got signal 11;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly built,
> or misconfigured. This error can also be caused by malfunctioning hardware.
> We will try our best to scrape up some info that will hopefully help diagnose
> the problem, but since we have already crashed, something is definitely wrong
> and this may fail.
>
> GDB Back Trace (with some debug symbols missing):
>
> -bash-4.0$ gdb drizzled
> GNU gdb (GDB) Fedora (7.0.1-48.fc12)
> Copyright (C) 2009 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/drizzled...Reading symbols from /usr/lib/debug/usr/sbin/drizzled.debug...done.
> done.
>
> (gdb) run
> Starting program: /usr/sbin/drizzled
> [Thread debugging using libthread_db enabled]
> InnoDB: The InnoDB memory heap is disabled
> InnoDB: Mutexes and rw_locks use GCC atomic builtins.
> Program received signal SIGSEGV, Segmentation fault.
>
> (gdb) bt full
> #0 0x0000000000000000 in ?? ()
> No symbol table info available.
> #1 0x00000000006722ab in drizzled::my_error (nr=2, MyFlags=36) at drizzled/error.cc:140
> format = "Can't create file '%-.200s' (errno: %d)"
> args = {{gp_offset = 32, fp_offset = 48, overflow_arg_area = 0x7fffffffb1c0,
> reg_save_area = 0x7fffffffb0d0}}
> ebuff = "Can't create file '/var/lib/drizzle//.temporary/ibdDJXTn' (errno: 13)\000\000\000\350\340\377\367\377\177\000\000 \336\376\367\377\177\000\000K\214\336\367\377\177\000\000\000\000\000\000\000\000\000\000 \336\376\367\377\177\000\000\001", '\000' <repeats 15 times>, "\001\000\000\000\000\000\000\000\350\340\377\367\377\177\000\000\b\000\000\000\000\000\000\000@\253\264\000\000\000\000\...

[Andrew Hutchings (linuxjedi)]

hmm... this will be harder than I thought. Did a quick test case (pushed to the branch for this bug) and forgot we now intentionally fail instead of segfault:

CURRENT_TEST: main.bug597905
One can only use the --user switch if running as root
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins.
/home/linuxjedi/Programming/Sources/drizzle-bug-597905/drizzled/.libs/lt-drizzled: Can't create file '/home/linuxjedi/Programming/Sources/drizzle-bug-597905/tests/var/tmp/.temporary/ib2DW5Az' (errno: 13)100710 23:51:30 InnoDB: Error: unable to create temporary file; errno: 13
Plugin 'InnoDB' init function returned error.
Unknown/unsupported storage engine: innodbAborting

Brian's way may be best so we can catch the exit code or at least log for drizzled and process that.

[Andrew Hutchings (linuxjedi)]

Test case creation blocked by bug #616870

[Andrew Hutchings (linuxjedi)]

Patrick to migrate some mtr2 features so that we can build a test case for this