Change buildout to not re-build libmemcached

[Paul Everitt]

We're not going to do all the work originally described below. Instead, for now, we're only going to take libmemached out of the buildout and rely on it being in the system. This might mean an update to pylibmc.

#####

Our production (and staging) updates now take too long. It's long enough to warrant some initial investigation. For this task:

- Watch the next karlstaging build/run
- Identify some low-hanging fruit on what is taking a long time
- Give some recommendations on speeding up

As an example, I believe we recompile memcache (or redis?) on each of the four VMs, each time we update. Is there a lib that is missing from the gocept host environment? Or perhaps we're pointing to a private copy?

Other ideas:

- Are we going to our private package repo on github for everything? Can we reduce network latency by using pypi.gocept.net?

Don't spend more than an hour on this.

[Paul Everitt (paul-agendaless)]

I will take this one back and consider whether we bump pylibmc and use a system libmemcached.

[Paul Everitt (paul-agendaless)]

Re-opening this. Here is the result of the previous, failed attempt:

In December, gocept rolled out a gentoo update that provided a new libmemcached. We wanted to stop building it in our buildouts and use theirs. Which mean a pylibmc update, as our old version couldn’t handle a new libmemcached.

On Dec 26, I updated to move to gocept’s Python 2.7, their system libmemcached, and pylibmc 1.4.1. All seemed well. But I didn’t know that, almost immediately, we were getting segfaults on file uploads. I didn’t know because Python didn’t have a chance to log before dying.

Here’s the log file dump (with a few more hundred lines after).

2014-12-26 11:58:36,277 WARNI [relstorage.pylibmc_wrapper][Dummy-2] get_multi failed: Failed to decompress value: realloc

*** Error in `/srv/osfkarl/production/63/bin/python2.6': double free or corruption (!prev): 0x9d6cf348 ***
======= Backtrace: =========
/lib/libc.so.6(+0x79422)[0xb741d422]
/lib/libc.so.6(+0x7a180)[0xb741e180]
/srv/osfkarl/production/63/eggs/pylibmc-1.4.1-py2.6-linux-i686.egg/_pylibmc.so(+0x381e)[0xb6dd581e]

I am reverting to the older combination of Python, self-build libmemcached, and older pylibmc.

[Chris McDonough (chrism-plope)]

On another project I used a combination of libmemcached-1.0.16 and pylibmc-1.3.0 successfuly (on CentOS 6). There's no thought put into upgrading to latest and greatest here, I just happen to know both of these work together. I have logged in to karlstaging and both libmemcached-1.0.16 and pylibmc-1.3.0 seem to build OK there. I think what I'm going to try to do is:

- Add code to osideploy which puts a version of libmemcached-1.0.16 into /srv/{whatever}/opt/libmemcached-1.0.16
   via "./configure --prefix=/srv/{whatever}/opt/libmemcached-1.0.16 --without-memcached" if it doesn't
  already exist there.

- Remove the cmmi recipe for libmemcached from the buildout.cfg generated by osideploy app server template.

- Change the recipe for the pylibmc egg in the buildout.cfg generated by osideploy app server template to use an
  -rpath of "/srv/{whatever}/opt/libmemcached-1.0.16" and update the version of pylibmc in the version pins to 1.3.0.

[Chris McDonough (chrism-plope)]

I think I need some help from whomever created "pylibmc-1.1.1-agendaless1" to see if the patch applied in our forked version is still necessary for pylibmc-1.3.0:

Here's the patch:

[chrism@thinko pylibmc-1.1.1]$ diff -u -r . ~/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/
diff -u -r ./PKG-INFO /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/PKG-INFO
--- ./PKG-INFO 2010-06-07 10:04:04.000000000 -0400
+++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/PKG-INFO 2011-04-06 16:20:58.000000000 -0400
@@ -1,6 +1,6 @@
 Metadata-Version: 1.0
 Name: pylibmc
-Version: 1.1.1
+Version: 1.1.1-agendaless1
 Summary: Quick and small memcached client for Python
 Home-page: http://sendapatch.se/projects/pylibmc/
 Author: Ludvig Ericson
diff -u -r ./_pylibmcmodule.c /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/_pylibmcmodule.c
--- ./_pylibmcmodule.c 2010-06-03 15:13:45.000000000 -0400
+++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/_pylibmcmodule.c 2011-04-06 16:48:29.000000000 -0400
@@ -1054,6 +1054,7 @@
      */

     memcached_return rc;
+ *err_func = NULL;

     rc = memcached_mget(mc, (const char **)keys, key_lens, nkeys);

@@ -1075,9 +1076,10 @@
         /* if loop spins out of control, this fails */
         assert(nkeys >= (*nresults));

+ res->key_len = 0;
         res->value = memcached_fetch(mc, res->key, &res->key_len,
                                      &res->value_len, &res->flags, &rc);
- assert(res->value_len < MEMCACHED_MAX_KEY);
+ assert(res->key_len < MEMCACHED_MAX_KEY);

         if (res->value == NULL || rc == MEMCACHED_END) {
             /* This is how libmecached signals EOF. */
diff -u -r ./pylibmc-version.h /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/pylibmc-version.h
--- ./pylibmc-version.h 2010-06-07 09:23:30.000000000 -0400
+++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/pylibmc-version.h 2011-04-06 16:22:03.000000000 -0400
@@ -1 +1 @@
-#define PYLIBMC_VERSION "1.1.1"
+#define PYLIBMC_VERSION "1.1.1-agendaless1"

[Paul Everitt (paul-agendaless)]

Given that putting thought into it was actually the problem last time (too ambitious on the combination), I agree to go with a combination you have confidence in.

Those bullets look fine. It's changing the buildout in osideploy so it won't affect developer buildouts, which is good. Thanks!

--Paul

> On Mar 18, 2015, at 1:46 PM, Chris McDonough <email address hidden> wrote:
>
> On another project I used a combination of libmemcached-1.0.16 and
> pylibmc-1.3.0 successfuly (on CentOS 6). There's no thought put into
> upgrading to latest and greatest here, I just happen to know both of
> these work together. I have logged in to karlstaging and both
> libmemcached-1.0.16 and pylibmc-1.3.0 seem to build OK there. I think
> what I'm going to try to do is:
>
> - Add code to osideploy which puts a version of libmemcached-1.0.16 into /srv/{whatever}/opt/libmemcached-1.0.16
> via "./configure --prefix=/srv/{whatever}/opt/libmemcached-1.0.16 --without-memcached" if it doesn't
> already exist there.
>
> - Remove the cmmi recipe for libmemcached from the buildout.cfg
> generated by osideploy app server template.
>
> - Change the recipe for the pylibmc egg in the buildout.cfg generated by osideploy app server template to use an
> -rpath of "/srv/{whatever}/opt/libmemcached-1.0.16" and update the version of pylibmc in the version pins to 1.3.0.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1352745
>
> Title:
> Change buildout to not re-build libmemcached
>
> Status in OSF KARL4:
> Confirmed
>
> Bug description:
> We're not going to do all the work originally described below.
> Instead, for now, we're only going to take libmemached out of the
> buildout and rely on it being in the system. This might mean an update
> to pylibmc.
>
> #####
>
> Our production (and staging) updates now take too long. It's long
> enough to warrant some initial investigation. For this task:
>
> - Watch the next karlstaging build/run
> - Identify some low-hanging fruit on what is taking a long time
> - Give some recommendations on speeding up
>
> As an example, I believe we recompile memcache (or redis?) on each of
> the four VMs, each time we update. Is there a lib that is missing
> from the gocept host environment? Or perhaps we're pointing to a
> private copy?
>
> Other ideas:
>
> - Are we going to our private package repo on github for everything?
> Can we reduce network latency by using pypi.gocept.net?
>
> Don't spend more than an hour on this.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl4/+bug/1352745/+subscriptions

[Paul Everitt (paul-agendaless)]

Oh brother, I didn't even know about that. Yikes. I suspect it was Shane?

--Paul

> On Mar 18, 2015, at 2:41 PM, Chris McDonough <email address hidden> wrote:
>
> I think I need some help from whomever created
> "pylibmc-1.1.1-agendaless1" to see if the patch applied in our forked
> version is still necessary for pylibmc-1.3.0:
>
> Here's the patch:
>
> [chrism@thinko pylibmc-1.1.1]$ diff -u -r . ~/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/
> diff -u -r ./PKG-INFO /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/PKG-INFO
> --- ./PKG-INFO 2010-06-07 10:04:04.000000000 -0400
> +++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/PKG-INFO 2011-04-06 16:20:58.000000000 -0400
> @@ -1,6 +1,6 @@
> Metadata-Version: 1.0
> Name: pylibmc
> -Version: 1.1.1
> +Version: 1.1.1-agendaless1
> Summary: Quick and small memcached client for Python
> Home-page: http://sendapatch.se/projects/pylibmc/
> Author: Ludvig Ericson
> diff -u -r ./_pylibmcmodule.c /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/_pylibmcmodule.c
> --- ./_pylibmcmodule.c 2010-06-03 15:13:45.000000000 -0400
> +++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/_pylibmcmodule.c 2011-04-06 16:48:29.000000000 -0400
> @@ -1054,6 +1054,7 @@
> */
>
> memcached_return rc;
> + *err_func = NULL;
>
> rc = memcached_mget(mc, (const char **)keys, key_lens, nkeys);
>
> @@ -1075,9 +1076,10 @@
> /* if loop spins out of control, this fails */
> assert(nkeys >= (*nresults));
>
> + res->key_len = 0;
> res->value = memcached_fetch(mc, res->key, &res->key_len,
> &res->value_len, &res->flags, &rc);
> - assert(res->value_len < MEMCACHED_MAX_KEY);
> + assert(res->key_len < MEMCACHED_MAX_KEY);
>
> if (res->value == NULL || rc == MEMCACHED_END) {
> /* This is how libmecached signals EOF. */
> diff -u -r ./pylibmc-version.h /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/pylibmc-version.h
> --- ./pylibmc-version.h 2010-06-07 09:23:30.000000000 -0400
> +++ /home/chrism/projects/karl/karlhosting.github.com/pyramid/production/pylibmc-1.1.1-agendaless1/pylibmc-version.h 2011-04-06 16:22:03.000000000 -0400
> @@ -1 +1 @@
> -#define PYLIBMC_VERSION "1.1.1"
> +#define PYLIBMC_VERSION "1.1.1-agendaless1"
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1352745
>
> Title:
> Change buildout to not re-build libmemcached
>
> Status in OSF KARL4:
> Confirmed
>
> Bug description:
> We're not going to do all the work originally described below.
> Instead, for now, we're only going to take libmemached out of the
> buildout and rely on it being in the system. This might mean an update
> to pylibmc.
>
> #####
>
> Our production (and staging) updates now take too long. It's long
> enough to warrant some initial investigation. For this task:
>
> - Watch ...

[Paul Everitt (paul-agendaless)]

Chris, do you think you will be doing any work on this? If not, I'll assign it to myself and look into it later.

[Chris McDonough (chrism-plope)]

I can keep it on my plate. The blocker (which is mostly mental at this point) is trying it out on a Gocept Gentoo system, and making sure it still works in development on Ubuntu 12/14 and Mac.