Valgrind SSL errors / possible leak with mysql-test-run

[Raghavendra D Prabhu]

1) Following are some of the valgrind errors I got while doing mysql-test-run for lp:890404 (since mysql-test-run has ssl in its default config):

 ==13145== Conditional jump or move depends on uninitialised value(s)
 ==13145== at 0x5DB867A: BN_bin2bn (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5DBCBB0: bnrand.part.0 (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E0A3AE: generate_key (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AD0DCD: ssl3_ctx_ctrl (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C815: new_VioSSLFd (viosslfactories.c:241)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)
 ==13145==

Use of uninitialised value of size 8
 ==13145== at 0x5DB7F8D: BN_num_bits_word (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5DB80D5: BN_num_bits (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5DB5BAB: BN_mod_exp_mont_consttime (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E0A3E7: generate_key (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AD0DCD: ssl3_ctx_ctrl (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C815: new_VioSSLFd (viosslfactories.c:241)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== Use of uninitialised value of size 8
 ==13145== at 0x5DC2D32: bn_mul4x_mont_gather5 (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E0A3E7: generate_key (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AD0DCD: ssl3_ctx_ctrl (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C815: new_VioSSLFd (viosslfactories.c:241)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== Use of uninitialised value of size 8
 ==13145== at 0x5DC2F8B: bn_mul4x_mont_gather5 (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xF4D04E9BCB457FC7: ???
 ==13145== by 0x9B02B665E7FAD2F4: ???
 ==13145== by 0x14E562FC4E6844AC: ???
 ==13145== by 0x3AB78E18ED6E0269: ???
 ==13145== by 0x17E826CB34C20B61: ???
 ==13145== by 0xDFEFB8996F51E6D4: ???
 ==13145== by 0x144946AE94BE6510: ???
 ==13145== by 0x24639DFE28B16EE: ???

 ==13145== Thread 1:
 ==13145== 24 bytes in 1 blocks are still reachable in loss record 1 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7A095: def_get_class (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7AC7A: int_new_ex_data (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E1550E: BIO_set (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E15599: BIO_new (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E17902: BIO_new_file (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E67133: X509_load_cert_crl_file (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6726C: by_file_ctrl (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E5DBAD: X509_STORE_load_locations (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C457: new_VioSSLFd (viosslfactories.c:216)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 3 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7A095: def_get_class (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7AC7A: int_new_ex_data (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E42E13: x509_cb (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E46EE0: asn1_item_ex_combine_new (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E49CC8: ASN1_item_ex_d2i (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E4A763: ASN1_item_d2i (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E56383: PEM_X509_INFO_read_bio (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6714F: X509_load_cert_crl_file (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6726C: by_file_ctrl (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E5DBAD: X509_STORE_load_locations (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C457: new_VioSSLFd (viosslfactories.c:216)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20EA5: lh_insert (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7A0C7: def_get_class (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7AC7A: int_new_ex_data (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E42E13: x509_cb (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E46EE0: asn1_item_ex_combine_new (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E49CC8: ASN1_item_ex_d2i (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E4A763: ASN1_item_d2i (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E56383: PEM_X509_INFO_read_bio (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6714F: X509_load_cert_crl_file (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6726C: by_file_ctrl (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E5DBAD: X509_STORE_load_locations (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C457: new_VioSSLFd (viosslfactories.c:216)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 5 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20EA5: lh_insert (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7BC58: OBJ_NAME_add (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E2DA64: EVP_add_cipher (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AF8A10: SSL_library_init (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C2FB: new_VioSSLFd (viosslfactories.c:151)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 6 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20EA5: lh_insert (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7BC58: OBJ_NAME_add (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AF8A10: SSL_library_init (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C2FB: new_VioSSLFd (viosslfactories.c:151)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 76 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7A095: def_get_class (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7A187: int_get_new_index (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E81DD5: COMP_zlib (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AF08F8: load_builtin_compressions (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0x5AF2E08: SSL_COMP_get_compression_methods (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0x5AF8BF5: SSL_library_init (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C2FB: new_VioSSLFd (viosslfactories.c:151)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 80 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7BC35: OBJ_NAME_add (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E31F00: OpenSSL_add_all_ciphers (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E31EDD: OPENSSL_add_all_algorithms_noconf (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C300: new_VioSSLFd (viosslfactories.c:152)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 24 bytes in 1 blocks are still reachable in loss record 79 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20EA5: lh_insert (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5D7BC58: OBJ_NAME_add (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E2DA64: EVP_add_cipher (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E31F00: OpenSSL_add_all_ciphers (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E31EDD: OPENSSL_add_all_algorithms_noconf (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C300: new_VioSSLFd (viosslfactories.c:152)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)
 ==13145==

 ==13145== 176 bytes in 1 blocks are still reachable in loss record 678 of 752
 ==13145== at 0x4C2BFCB: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78A4F: CRYPTO_malloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20A6F: lh_new (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E235FE: int_thread_get (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E237FB: int_thread_set_item (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E24E54: ERR_get_state (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E25D8E: ERR_put_error (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E5806D: PEM_read_bio (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E560CE: PEM_X509_INFO_read_bio (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6714F: X509_load_cert_crl_file (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E6726C: by_file_ctrl (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E5DBAD: X509_STORE_load_locations (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0xA3C457: new_VioSSLFd (viosslfactories.c:216)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)

 ==13145== 16,384 bytes in 1 blocks are still reachable in loss record 738 of 752
 ==13145== at 0x4C2C1DE: realloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
 ==13145== by 0x5D78B60: CRYPTO_realloc (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E20E31: lh_insert (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E23E0D: int_err_set_item (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5E24458: ERR_load_strings (in /usr/lib/libcrypto.so.1.0.0)
 ==13145== by 0x5AF99BD: ERR_load_SSL_strings (in /usr/lib/libssl.so.1.0.0)
 ==13145== by 0xA3C315: new_VioSSLFd (viosslfactories.c:159)
 ==13145== by 0xA3C91E: new_VioSSLAcceptorFd (viosslfactories.c:288)
 ==13145== by 0x51D509: mysqld_main(int, char**) (mysqld.cc:3735)
 ==13145== by 0x513974: main (main.cc:25)
 ==13145==

2) I will upload the complete file along with generated suppressions.

3) Most (or all) of these are false positives:

      http://<email address hidden>/msg22467.html
      https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/60021
      http://www.openssl.org/support/faq.html#PROG13

So, we will need to update the valgrind.supp under mysql-test/ with these. We can also import these suppressions from any other project which uses openssl I guess.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Complete file with valgrind errors and generated suppressions.

[Raghavendra D Prabhu (raghavendra-prabhu)]

I invoked valgrind as:

./mysql-test-run.pl --valgrind --valgrind-option="--suppressions=$PWD/valgrind.supp" --valgrind-option='--show-reachable=yes' --valgrind-option='--gen-suppressions=all' --vardir=$HOME/mysql t/fake.test (fake.test is the test used there but any test should do since ssl is in default mtr config)

[Laurynas Biveinis (laurynas-biveinis)]

Does upstream have these errors?

[Raghavendra D Prabhu (raghavendra-prabhu)]

I haven't tested with upstream yet, but I checked the suppression file valgrind.supp and it is the same.

[Laurynas Biveinis (laurynas-biveinis)]

It could be because if the upstream builds using slightly different build configuration and that the upstream will produce the same errors built in our way.

[Roel Van de Paar (roel11)]

Sidenote: http://lists.mysql.com/commits/145376

[Raghavendra D Prabhu (raghavendra-prabhu)]

Yes, according to that, it looks like it may not require a suppression addition for Valgrind but an actual fix. It also looks like the suppression were added originally only in 5.6.

[Raghavendra D Prabhu (raghavendra-prabhu)]

The upstream bug for this seems to be a private one - 15908967 -- VALGRIND ERRORS WHEN RUNNING WITH OPENSSL

[Raghavendra D Prabhu (raghavendra-prabhu)]

This is from vio/vio.c: (in contrast to the patch here http://lists.mysql.com/commits/145376)

void vio_end(void)
{
#ifdef HAVE_YASSL
  yaSSL_CleanUp();
#endif
}

Also, since we have switched to openssl, is HAVE_YASSL still valid?

[Roel Van de Paar (roel11)]

See bug 1191582 & bug 1205196

[Roel Van de Paar (roel11)]

5.6 Valgrind testing;

==2838== 32 bytes in 1 blocks are still reachable in loss record 1 of 437
==2838== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==2838== by 0x364265D9CD: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.0)
==2838== by 0x36426B6F5E: sk_new (in /usr/lib64/libcrypto.so.1.0.0)
==2838== by 0x364663D0E4: ??? (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0x364663D308: SSL_COMP_get_compression_methods (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0x3646644651: SSL_library_init (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0xC8B4EB: ssl_start (viosslfactories.c:155)
==2838== by 0x629359: init_ssl() (mysqld.cc:4323)
==2838== by 0x634076: mysqld_main(int, char**) (mysqld.cc:5525)
==2838== by 0x3636E1ECDC: (below main) (in /lib64/libc-2.12.so)
==2838== 32 bytes in 1 blocks are still reachable in loss record 2 of 437
==2838== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==2838== by 0x364265D9CD: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.0)
==2838== by 0x36426B6F7C: sk_new (in /usr/lib64/libcrypto.so.1.0.0)
==2838== by 0x364663D0E4: ??? (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0x364663D308: SSL_COMP_get_compression_methods (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0x3646644651: SSL_library_init (in /usr/lib64/libssl.so.1.0.0)
==2838== by 0xC8B4EB: ssl_start (viosslfactories.c:155)
==2838== by 0x629359: init_ssl() (mysqld.cc:4323)
==2838== by 0x634076: mysqld_main(int, char**) (mysqld.cc:5525)
==2838== by 0x3636E1ECDC: (below main) (in /lib64/libc-2.12.so)

Can we filter this one if benign? It causes runs to fail into STATUS_VALGRIND_ERROR

[Laurynas Biveinis (laurynas-biveinis)]

Sergei -

Can you investigate?

[Laurynas Biveinis (laurynas-biveinis)]

Half of the messages are duplicate of bug 1191582, and half of bug 1205196.