Embedded server asserts on startup because of pbxt

[Sergey Petrunia]

Attempt to run any test with --embedded server will fail becasue of pbxt in the following way:

./mysql-test-run --embedded-server t/alias.test
"our" variable $opt_suites masks earlier declaration in same scope at ./mysql-test-run line 134.
Logging: ./mysql-test-run --embedded-server t/alias.test
091001 14:22:08 [Note] Plugin 'FEDERATED' is disabled.
MySQL Version 5.1.38
Checking supported features...
 - skipping ndbcluster
 - skipping SSL
 - binaries are debug compiled
Collecting tests...
vardir: /home/psergey/bzr-new/mysql-5.1-maria-contd4/mysql-test/var
Checking leftover processes...
Removing old var directory...
Creating var directory '/home/psergey/bzr-new/mysql-5.1-maria-contd4/mysql-test/var'...
Installing system database...
Using server port 52935

==============================================================================

TEST RESULT TIME (ms)
------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 13000..13009
091001 13:22:17 [Note] PrimeBase XT (PBXT) Engine 1.0.08d RC loaded...
091001 13:22:17 [Note] Paul McCullagh, PrimeBase Technologies GmbH, http://www.primebase.org
main.alias [ fail ]
        Test ended at 2009-10-01 14:22:17

CURRENT_TEST: main.alias
thd->thread_stack=0x1 addr is 0xa6f80b4
thd=0xa6f7790 size=7092
mysqltest_embedded: sql_class.cc:1136: bool THD::store_globals(): Assertion `thread_stack' failed.
mysqltest got signal 6
read_command_buf at 0x89df0a0 =
Attempting backtrace...
thd->thread_stack=0x1 addr is 0xa6f9e04
thd=0xa6f94e0 size=7092
mysqltest_embedded: sql_class.cc:1136: bool THD::store_globals(): Assertion `thread_stack' failed.

 - saving '/home/psergey/bzr-new/mysql-5.1-maria-contd4/mysql-test/var/log/main.alias/' to '/home/psergey/bzr-new/mysql-5.1-maria-contd4/mysql-test/var/log/main.alias/'
------------------------------------------------------------
The servers were restarted 0 times
Spent 0.000 of 15 seconds executing testcases

Failed 1/1 tests, 0.00% were successful.

Failing test(s): main.alias

[Sergey Petrunia (sergefp)]

My analysis shows that PBXT (myxt_create_thread() in storage/pbxt/src/myxt_xt.cc in particular) thinks that sizeof(THD) is 7092,
while the SQL layer (THD::store_globals() in particular) thinks that sizeof(THD) is 6880).

When myxt_create_thread() will make this assignment:

  new_thd->thread_stack = (char *) &new_thd;

it will "miss" the thread_stack variable and set value of something else. THD::store_globals(), which is called immediately after, will still see thread_stack==0 and fail an assertion.

The reason for sizeof difference seems to be that THD has different sizes depending on whether it is an embedded server or not. PBXT has sizeof(THD) from non-embedded server and hence fails in embedded server.

[Sergey Petrunia (sergefp)]

Possible solutions:

1. Disable PBXT in embedded build (but how exactly can one do that?)

2. Let PBXT be indepdendent of sizeof(THD): let SQL layer provide a factory function for THD and accessor functions for whatever members PBXT needs.

[Paul McCullagh (paul-mccullagh)]

Hi Sergey,

According to your analysis, PBXT is being compiled with the wrong options. I am wondering, when MySQL is built, does it:

1) build 2 objects for each source file: one for mysqld and the other for the server embedded library?

2) or does it use the same object to link both mysqld and the embedded library?

[Sergey Petrunia (sergefp)]

Hi Paul,

For SQL layer, #1 is true. It does build two objects for each source file. The regular server is compiled in sql/ while embedded server is in libmysqld/ (which is populated with symbolic links to sql/*.cc).

For storage engines, #1 is used by default.

There are storage engines that have some files that are dependent on server's internals and they've got a plug.in file directive named
 MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS which tells the build system that it needs to build two different copies, one for regular server and one for embedded.

For example, storage/heap/plug.in has:

MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(heap, [ha_heap.cc])

I've grepped for 'thd->' in PBXT source code, which hinted at the need to add:

MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(pbxt, [src/ha_pbxt.cc])
MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(pbxt, [src/myxt_xt.cc])
MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(pbxt, [src/discover_xt.cc])

The problem is that that doesn't work because it looks like at the moment each plugin can only have one MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS clause (see http://lists.mysql.com/internals/37339).

Does PBXT really need to depend so much on server internals? (from three files. other storage engines, e.g. InnoDB all have one dependant file).

[Paul McCullagh (paul-mccullagh)]

Hi Sergey,

Unfortunately PBXT does rely a lot on MySQL internals.

It is a small thing which has wide consequences.

PBXT uses the MySQL internal data dictionary. Not only for information
(which could be easily duplicated), but also for functionality. In
particular, sorting of strings according to collation sequences,
unicode, rtf8, etc.

So to avoid having to re-write or duplicate this code in the engine,
PBXT requires accesses to the internal TABLE structure used by MySQL.

This is normally no problem for an engine that only accesses tables in
handler calls (for example MyISAM).

But there are 2 instances in which the is not the case for PBXT:

1. During recovery
2. The background threads.

Both the recovery process, and the background threads require a TABLE
reference when accessing tables.

If the table has not yet been opened then PBXT is required to open a
MySQL TABLE structure.

This in turn means that the recovery process and the background
threads have to create a THD structure (because you can't open a TABLE
structure in MySQL without one).

So this, in turn, leads to some hacking, because there are no
interfaces in MySQL to do this.

I suggest the following approach to this problem:

1. If possible, we should disable PBXT in the runtime context. Do you
know of a runtime variable that indicates the code is running in
runtime mode? This can be used to prevent any PBXT processes from
starting. In this way we can prevent crashes until we can figure out
how to get PBXT to work in the runtime environment.

2. Next we should prevent the PBXT recovery process and the background
threads from starting when there are no PBXT tables (or at least no
PBXT system database).

This used to be the case, but since the change to the "global
database", this functionality was lost. I am not sure how difficult
this is, so I will have to take a look.

3. Either we need to fix MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(), or
we need to move all MySQL dependencies in PBXT into one file.

However, if by dependencies we mean all references to MySQL internal
structures, then this would mean moving ALL code in myxt_xt.cc and
discover_xt.cc into ha_pbxt.cc. I would be reluctant to do this.

If it is just references to THD, then this is not quite so bad.

Best regards,

Paul

On Oct 1, 2009, at 11:15 PM, Sergey Petrunia wrote:

> Hi Paul,
>
> For SQL layer, #1 is true. It does build two objects for each source
> file. The regular server is compiled in sql/ while embedded
> server is
> in libmysqld/ (which is populated with symbolic links to sql/*.cc).
>
> For storage engines, #1 is used by default.
>
> There are storage engines that have some files that are dependent on
> server's internals and they've got a plug.in file directive named
> MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS which tells the build system
> that it needs to build two different copies, one for regular server
> and one for embedded.
>
> For example, storage/heap/plug.in has:
>
> MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS(heap, [ha_heap.cc])
>
> I've grepped for 'thd->' in PBXT source code, which hinted at the
> need
> to add:
>
> MYSQL_PLUGIN_DE...

[Hakan Küçükyılmaz (hakan-askmonty)]

Sergey, can you please check how we can fix this issue? Can you also adjust the "Importance" field of this bug report, please?

Thanks

[Vladimir Kolesnikov (vkolesnikov)]

The original problem happened because pbxt had dependencies on mysql structures in several sources but macro MYSQL_PLUGIN_DEPENDS_ON_MYSQL_INTERNALS did not work for > 1 files. So some of the dependent source were built with wrong options. After fixing this problem wrong object files were still used. This was because of the following: libmysqld build script extracts object files from all plugin libraries and puts them into a single temporary archive. Next, object code of the files that depend on mysql internals are put into that same archive in replacement mode, so that if we already have e.g. ha_pbxt.o (compiled for normal server) in the archive then it will be replaced with the correct version built for embedded server. The problem was that pbxt used CXXFLAGS automake variable in its Makefile.am and in such case automake automatically adds target-specific prefix to the output object file to avoid conflicts when the same source is built several times with different options. In our case we got something like libpbxt_a-ha_pbxt.o. The result was that the old ha_pbxt.o was not replaced in the archive and linker used symbols from there (by a chance I guess). The solution was not to use CXXFLAGS (we didn't really need them) but this is an undocumented limitation and might be a problem in future for other plugins too.