[SRU] Using `fastcgi_cache` or `proxy_cache` with nginx-extras causes the push module to throw errors.

[Arnaud]

[Impact]

 * The nginx-http-push module generates errors when using fastcgi_cache or proxy_cache arguments in the config.

 * This adds extra errors to the error logs, and causes problems with the cache manager.

 * Caching still works, however, but this causes excessive unnecessary noise in the error logs and the issue is an incompatibility between the module and the nginx cache manager.

[Test Case]

 * Use the nginx-extras package, and use any configuration with either `fastcgi_cache` or `proxy_cache` arguments. (NOTE: I have provided a test case below that works with php5-fpm and this. You do not need php5-fpm for this to work, however it would not hurt to have it installed at the same time.)

[Regression Potential]

 * This fix originated upstream, and should not introduce any regressions.

[Other Info]

 * The debdiffs here are derived from upstream changes, as detailed at https://github.com/slact/nginx_http_push_module/commit/836e8319c93681386fb00e6bd34d9e37612f3334.diff

 * The only changes to this diff were the indentation, to match what was in the packages.

[Original Description]

system : Ubuntu 12.04 LTS
package : nginx-extras 1.4.1-1ppa1~precise

Hello,
if you try to use proxy_cache or fastcgi_cache you got this error in error.log of nginx :

2013/08/26 00:13:10 [info] 15722#0: Using 32KiB of shared memory for push module in /etc/nginx/nginx.conf:137
2013/08/26 00:13:11 [alert] 15756#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/08/26 00:13:11 [alert] 15756#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/08/26 00:13:11 [alert] 15757#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/08/26 00:13:11 [alert] 15757#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/08/26 00:13:11 [alert] 15737#0: cache manager process 15756 exited with fatal code 2 and cannot be respawned

You can't disable nginx_push_module, to make nginx cache to work.
Everything is fine if you use the "daemon off;" option.

I've seen this issue in many forum/mailing list, i think nginx_push_module shouldn't be in the extras package anymore.

system : Ubuntu 12.04 LTS
package : nginx-extras 1.4.1-1ppa1~precise

[Joe Clifford (joeclifford)]

Confirmed. I'm seeing this issue also on nginx-extras 1.1.19-1ubuntu0.4

[Thomas Ward (teward)]

Arnaud, are you still getting this with 1.4.3? If so, I will consult with Debian about this, as the packaging in the PPA is pretty much directly from them.

------

Joe Clifford, please open a bug against the Ubuntu package specifically, as that needs to be handled differently from this bug.

[Thomas Ward (teward)]

Reported to be seen in Saucy as well.

[Thomas Ward (teward)]

This is a two part bug: This bug has been reported to be present in NGINX team's PPAs as well as the version(s) in the Ubuntu repositories.

[Thomas Ward (teward)]

To all impacted by this bug: Please test the version in Trusty (1.4.3) either with the nginx team's PPA or by testing on Trusty itself (in a VM or otherwise). I need to know whether 1.4.3 is impacted by this bug.

[Joe Clifford (joeclifford)]

I've just quickly installed nginx-extras and nginx-common 1.4.3 from the
nginx PPA on to a 12.04 LTS VPS serving a couple of web sites. It is
running without showing those errors in the error.log any more which is
good news. I don't have time to dig too deep and test properly at the
moment but I'll leave the new nginx install running over the next few days
and let you know if anything goes wrong.

On 19 November 2013 16:47, Thomas Ward <email address hidden> wrote:

> To all impacted by this bug: Please test the version in Trusty (1.4.3)
> either with the nginx team's PPA or by testing on Trusty itself (in a VM
> or otherwise). I need to know whether 1.4.3 is impacted by this bug.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1216817
>
> Title:
> issue with nginx cache and nginx push module in nginx-extra package
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nginx/+bug/1216817/+subscriptions
>

[Arnaud (arnall)]

Hi,

using 1.4.3-2~precise0:

2013/11/28 13:00:59 [info] 2441#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:84
2013/11/28 13:01:00 [alert] 2475#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2475#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2476#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2476#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/11/28 13:01:00 [alert] 2453#0: cache manager process 2475 exited with fatal code 2 and cannot be respawned

But the fastcgi_cache seems to work...
I have cache files generated in the cache directory...
weird...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Juan Carlos Bauza Ogazón (jcbauza)]

I get the same error in an Ubuntu 13.10 machine running the latest mainline reléase:
2013/12/15 07:57:57 [info] 1621#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:11:27 [info] 1674#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:11:28 [alert] 1691#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1691#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1686#0: cache manager process 1691 exited with fatal code 2 and cannot be respawned
2013/12/15 08:11:28 [alert] 1692#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:11:28 [alert] 1692#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:22 [info] 1702#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:16:23 [alert] 1719#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1719#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1720#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1720#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:16:23 [alert] 1714#0: cache manager process 1719 exited with fatal code 2 and cannot be respawned
2013/12/15 08:21:48 [info] 1749#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:181
2013/12/15 08:21:49 [alert] 1767#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1767#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1766#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 08:21:49 [alert] 1766#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)2013/12/15 06:48:30 [info] 1155#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78

On a different computer were don't use proxy_cache I see only the info entry:
2013/12/15 06:59:01 [info] 1160#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 07:27:51 [info] 1213#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 07:37:24 [info] 1175#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:02:34 [info] 1799#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:02:42 [info] 1805#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:78
2013/12/15 08:21:49 [alert] 1761#0: cache manager process 1766 exited with fatal code 2 and cannot be respawned

[Juan Carlos Bauza Ogazón (jcbauza)]

I also enabled fast cgi cache on the computer that was working fine and now it is experiencing the same issue as with the computer that was using proxy cache:
2013/12/15 17:05:09 [info] 3939#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:87
2013/12/15 17:05:10 [alert] 3956#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3956#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3951#0: cache manager process 3956 exited with fatal code 2 and cannot be respawned
2013/12/15 17:05:10 [alert] 3957#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2013/12/15 17:05:10 [alert] 3957#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)

[Thomas Ward (teward)]

Can someone post an example of their configuration, so I can attempt to replicate this in Debian's packages? If this breakage also occurs in Debian then it can be fixed there, and depending on how they fix it, I may be able to SRU the updates for Ubuntu, but I will be able to fix the issues in the PPAs.

[Thomas Ward (teward)]

I have used a plain test example from Digital River on getting cache working with php5-fpm, and can replicate this error with `fastcgi_cache` in Debian. This is being forwarded to Debian for a fix, and I will attach a bug number here later.

[Thomas Ward (teward)]

This is reported in Debian against the nginx-extras package as Debian bug 732251. (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=732251)

As this has been sent up to Debian for a fix, because this is replicated in *their* packages first and foremost, I am marking both the Nginx project bug and the bug against the nginx packages in Ubuntu as "triaged", and the severity has been set to "Low" for both as this does not actually impact the ability for the system to cache data for `proxy_cache` or `fastcgi_cache` arguments, based on testing done by myself and one of the Debian maintainers.

[Leo P. (push-harder)]

Hello. I'm the writer of the push module. This is a problem with my code, not nginx, and I am working on a fix. See https://github.com/slact/nginx_http_push_module/issues/83 for the github discussion on this bug.

[Leo P. (push-harder)]

Fixed in version 0.711, available from https://github.com/slact/nginx_http_push_module/archive/v0.711.tar.gz . checksums at https://pushmodule.slact.net

[Vorobyoff Andrey (avorobyoff)]

Hello!
I'm using Ubuntu Server 14.04.1 in VM

# cat /etc/issue
Ubuntu 14.04.1 LTS \n \l
# uname -a
Linux vm-newsru-fe-02 3.13.0-35-generic #62-Ubuntu SMP Fri Aug 15 01:58:42 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I'm using nginx
# dpkg -l | grep nginx
ii nginx-common 1.4.6-1ubuntu3.1 all small, powerful, scalable web/proxy server - common files
ii nginx-extras 1.4.6-1ubuntu3.1 amd64 nginx web/proxy server (extended version)

I use nginx for proxying with cache.
Cache size set to 1024M but it is growing much more because nginx cache manager do not work:

# tail /var/log/nginx/error.log
2014/10/23 07:47:47 [info] 11822#0: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:109
2014/10/23 07:47:49 [alert] 11840#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11840#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11839#0: epoll_ctl(1, 0) failed (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11839#0: failed to register channel handler while initializing push module worker (1: Operation not permitted)
2014/10/23 07:47:49 [alert] 11827#0: cache manager process 11839 exited with fatal code 2 and cannot be respawned

[Thomas Ward (teward)]

This should be fixed now that the packages have been resynced with Debian which I believe had an updated version of the module included in it.

The package in Ubuntu is still affected.

[Thomas Ward (teward)]

Marking as "Fix Released" for other releases - this was fixed in Debian as 1.4.7-1, and Utopic and later have 1.6.x. As such, this is fixed in Utopic and later.

[Thomas Ward (teward)]

Upstream diff from the nginx-http-push module developers is located here: https://github.com/slact/nginx_http_push_module/commit/836e8319c93681386fb00e6bd34d9e37612f3334.diff

[Thomas Ward (teward)]

Precise debdiff for fixing this bug, using the upstream changes.

[Thomas Ward (teward)]

Trusty debdiff for fixing this bug

[Thomas Ward (teward)]

To everyone affected: We need example configurations to use here, so we can test this getting fixed.

To the SRU team: I am build-testing this now in a ppa, and will let you know if it builds correctly.

[Thomas Ward (teward)]

To the SRU team: The changes outlined in the debdiffs have successfully built on i386 and amd64, the two primary architectures. I do not have the alternate architectures available on my test PPA. https://launchpad.net/~teward/+archive/ubuntu/nginx-sru/+packages is the test PPA, with a focus on the details of the packages in the PPA.

[Vorobyoff Andrey (avorobyoff)]

Hello, Thomas!
I sent work nginx configuration to your e-mail (on ubuntu.com).

[Thomas Ward (teward)]

Vorobyoff: Thanks for providing the conf, that gives me the basic information I needed to start building a test configuration.

To all others: I estimate I'll have a test configuration available by the end of the day, and I will upload the test configuration here accordingly when finished.

[Thomas Ward (teward)]

Here's a test procedure:

(1) Use the attached test configuration file as a base, to test with localhost. (Partly derived from digital ocean's tutorials)

(2) Create a PHP file in the specified path. Alternatively, use an alternate root path.

(3) `tail -f /var/log/nginx/error.log`. Watch for any error outputs.

(4) Load nginx with the config. There should not be any errors triggered similar to above.

(5) To further test, load the PHP file you created. It should be rendered correctly and there should not be any errors in the error log similar to those observed with this bug.

[Thomas Ward (teward)]

Confirmed that this testcase works.

You may use ppa:teward/nginx-sru as a test build if you wish to really test this before it's uploaded to the proposed repositories.

I tested this on Trusty, but it should also work for precise.

REMEMBER: nginx-extras has the module so this needs to be tested with nginx-extras and not any other package.

[Thomas Ward (teward)]

Please use the test configuration for your OS version - the only difference is the fastcgi_pass line because the php5-fpm listener doesn't listen in the same place on Precise as it does in Trusty (and vice versa)

[Vorobyoff Andrey (avorobyoff)]

Hello!
Can anybody say when this bug will be fixed?
Now nginx-extras in 14.04 is in unusable state. We are stopping to use 14.04 in production.

[Thomas Ward (teward)]

Vorobyoff:

This will be fixed whenever the SRU team sees the upload - it's in the queue.

In the mean time, you can use my PPA with these changes already present - https://launchpad.net/~teward/+archive/ubuntu/nginx-sru/

It's ultimately the same builds that the debdiffs help make. If you can confirm the changes work that is one step in this - confirming the fixes actually work.

[Vorobyoff Andrey (avorobyoff)]

We use nginx with patches in production 2 days.
Everything is fine - no errors in logs, cache is not growing.

[Thomas Ward (teward)]

Removed old debdiffs because they're outdated and superseded by other updates. Also removed the sponsors team since I have upload rights now.

Thanks to Robie Basak for giving me a second-opinion technical review of the SRUs, before I work on pushing them up.

Updated the debdiffs and am reattaching them here.

[Thomas Ward (teward)]

Severity increased for Trusty, Precise, because in further testing I was able to replicate the comments from #16 where the cache can grow beyond the maximum set bounds.

[Chris J Arges (arges)]

Hello Arnaud, or anyone else affected,

Accepted nginx into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nginx/1.1.19-1ubuntu0.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Arnaud, or anyone else affected,

Accepted nginx into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nginx/1.4.6-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jered Brent (jeredbrent)]

Been pulling my hair out on this...

Rolled the trusty-proposed and it fixed the crazy logs.
From basic usage it looks like this is good on 14.04 x64
Upgraded both nginx_common & extras (1.4.6-1ubuntu3.2)

Thanks!

J

[Thomas Ward (teward)]

(Added trusty-verification-done per #38 which used trusty-proposed and confirmed this is fixed there.)

[Kristofer Pettijohn (kristofer)]

I can also confirm that trusty-proposed fixed this same issue for me.

[Thomas Ward (teward)]

Can someone test this on Precise as well? Precise needs to be tested to see if it is fixed with the package in proposed there.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nginx - 1.4.6-1ubuntu3.2

---------------
nginx (1.4.6-1ubuntu3.2) trusty-proposed; urgency=medium

  * d/modules/nginx-http-push: Apply upstream bugfix. (LP: #1216817)
    * src/ngx_http_push_module_setup.c: Modify push module code with
      upstream changes to fix an issue with initialization when using
      `fastcgi_cache` or `proxy_cache`.
    * tests/nginx-cachemanager.conf: (new file) Include upstream change
      of adding an nginx-cachemanager.conf file to the tests.
 -- Thomas Ward <email address hidden> Mon, 09 Feb 2015 12:08:50 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for nginx has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Thomas Ward (teward)]

Please note to everyone: This is only fixed in TRUSTY so far - we need Precise tests to be done with the package in precise-proposed to get it from precise-proposed to precise-updates

[Thomas Ward (teward)]

Tested in Precise and it removes the error output and problems as well for Precise.

[Launchpad Janitor (janitor)]

This bug was fixed in the package nginx - 1.1.19-1ubuntu0.8

---------------
nginx (1.1.19-1ubuntu0.8) precise-proposed; urgency=medium

  * d/modules/nginx-http-push: Apply upstream bugfix. (LP: #1216817)
    * src/ngx_http_push_module_setup.c: Modify push module code with
      upstream changes to fix an issue with initialization when using
      `fastcgi_cache` or `proxy_cache`.
    * tests/nginx-cachemanager.conf: (new file) Include upstream change
      of adding an nginx-cachemanager.conf file to the tests.
 -- Thomas Ward <email address hidden> Mon, 09 Feb 2015 12:02:52 -0500