Percona-Server-5.6 cmake compiler always check RocksDB even if we pass -DWITHOUT_ROCKSDB=1 argument

[Ramesh Sivaraman]

Percona-Server-5.6 cmake compiler always check RocksDB even if we pass -DWITHOUT_ROCKSDB=1 argument

How to repeat:

git clone https://github.com/percona/percona-server -b 5.6

cmake ../percona-server -DCMAKE_BUILD_TYPE=Debug -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DENABLE_DTRACE=OFF -DWITH_EXAMPLE_STORAGE_ENGINE=ON -DWITH_SSL=system -DWITH_EMBEDDED_SERVER=OFF -DWITHOUT_TOKUDB=ON -DENABLE_DOWNLOADS=1 -DWITHOUT_ROCKSDB=1

Compilation log:

$ cmake ../percona-server -DCMAKE_BUILD_TYPE=Debug -DBUILD_CONFIG=mysql_release -DFEATURE_SET=community -DENABLE_DTRACE=OFF -DWITH_EXAMPLE_STORAGE_ENGINE=ON -DWITH_SSL=system -DWITH_EMBEDDED_SERVER=OFF -DWITHOUT_TOKUDB=ON -DENABLE_DOWNLOADS=1 -DWITHOUT_ROCKSDB=1
-- Running cmake version 2.8.12.2
-- Found Git: /usr/bin/git (found version "1.7.1")
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
[..]
-- Performing Test HAVE_WVLA
-- Performing Test HAVE_WVLA - Success
CMake Error at storage/rocksdb/CMakeLists.txt:4 (MESSAGE):
  Missing Makefile in rocksdb directory. Try "git submodule update".

-- Not building with MyRocks storage engine, you must specify -DWITH_ROCKSDB=1 to cmake in order to build with MyRocks.
-- Looking for sched_getcpu
-- Looking for sched_getcpu - found
-- Performing Test HAVE_IB_GCC_ATOMIC_BUILTINS
-- Performing Test HAVE_IB_GCC_ATOMIC_BUILTINS - Success
[..]
See also "/home/ramesh/work/percona-server/CMakeFiles/CMakeOutput.log".
See also "/home/ramesh/work/percona-server/CMakeFiles/CMakeError.log".
$

[George Ormond Lorch III (gl-az)]

First, when building PS on any platform, one should _always_ either clone with --recursive option or do git submodule init && git submodule update after the clone.

Next, this is a simple issue in the rocksdb CMakeLists.txt where it shouldn't check for the existence of the lower rocksdb CMakeLists.txt before it checks if WITH_ROCKSDB is defined:

3 IF (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/rocksdb/Makefile")
4 MESSAGE(SEND_ERROR "Missing Makefile in rocksdb directory. Try \"git submodule update\".")
5 ENDIF()
6
7 IF (NOT WITH_ROCKSDB)
8 MESSAGE(STATUS "Not building with MyRocks storage engine, you must specify -DWITH_ROCKSDB=1 to cmake in order to build with MyRocks.")
9 RETURN()
10 ELSE ()
11 MESSAGE(STATUS "Building with MyRocks storage engine")
12 ENDIF ()

These should probably be swapped.

[Sergei Glushchenko (sergei.glushchenko)]

George,

I do not agree with your point re: git submodule init && git submodule update. If I don't need toku or rocks, why should I perform extra steps and download the code which won't even be compiled.

[Roel Van de Paar (roel11)]

I second that. Had we not seen this, some user/customer who self-compiles may have run into it.

[George Ormond Lorch III (gl-az)]

To be clear, the git submodules we use are only small parts of the overall source tree. TokuDB and its massive mtr suite comes along with the tree regardless of submodule, same will be for MyRocks. Just because _right_now_ we only use submodules for these external libraries that are pert of components that may be conditionally compiled in or out does not mean that submodule use will not be introduced or expanded elsewhere in the future. Specifically we have issues with overlapping imports of compression libraries. One of the possible (and perhaps likely) solutions is to import them as submodules for static compilation into a Percona third party repo which would pulled into Percona Server via a submodule. These compression libraries could/would also be utilized by other areas of the server (InnoDB column compression already had a request to try zstd) and thus would require the same process.

So yes, in order to have a truly complete working tree, git submodule init && update is required. Our jenkins templates all do it and they are our golden standard. This is why this issue was not caught by jenkins testing as one would have expected.

The use of submodules is actually intended to _improve_ the self compilation case. Rather than forcing a user to go out and grab code from several different locations and perfectly overlay them into the proper places in the source tree, we use the submodules instead. This is the way TokuDB used to be built. You had to manually copy code from 5 different external repos (IIRC they were tokutek/tokudb-engine, tokutek/percona-server, tokutek/ft-index, tokutek/enterprise-backup, and tokutek/backup-plugin) into several locations in the Percona Server source tree (6th repo) before you built. Now one single command 'git clone --recursive xxx' properly builds out everything needed to compile.

[Sergei Glushchenko (sergei.glushchenko)]

I am not saying using submodules is inconvenient, I am just saying that we should not require it if user won't build extras. Personally I almost never run this command because I don't usually compile tokudb on my laptop. When tokudb had issues with compiling on OS X, I simply put -DWITHOUT_TOKUDB=1 to my build script. It is saving some time for me every time I issue clean build as well.

[George Ormond Lorch III (gl-az)]

5.6
http://jenkins.percona.com/view/PS%205.6/job/percona-server-5.6-param/1464/
https://github.com/percona/percona-server/pull/1153
5.7
http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/491/
https://github.com/percona/percona-server/pull/1154

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1758