Bug #2069921 “gzip_static directive stopped working” : Bugs : nginx package : Ubuntu

Revision history for this message

Ovid Stavrica (pinnion) wrote on 2024-06-19:

#1

Dependencies.txt Edit (3.3 KiB, text/plain; charset="utf-8")
ProcCpuinfoMinimal.txt Edit (785 bytes, text/plain; charset="utf-8")

Revision history for this message

Paride Legovini (paride) wrote on 2024-06-20:

#2

Hello and thanks for your bug report. Looks like you are using

Package: nginx 1.18.0-6ubuntu14.4

from jammy-updated, but that update is from almost one year ago, and the (only) d/changelog entry looks very unrelated to what you are reporting, see [1].

Can you please share more information on the update history of your servers, and on what update specifically you believe caused the regression? Also: could you provide a minimal reproducer? Something on these lines:

  lxc launch ubuntu:jammy nginx-j
  lxc exec nginx-j bash
  apt install nginx
  # configure nginx
  curl [someoptions] http://localhost/...
  # X should be Y but is Z

Thanks!

[1] https://launchpad.net/ubuntu/+source/nginx/1.18.0-6ubuntu14.4

Changed in nginx (Ubuntu):
status:	New → Incomplete

Revision history for this message

Ovid Stavrica (pinnion) wrote on 2024-06-24: Re: [Bug 2069921] Re: gzip_static directive stopped working

#3

CgSLDMLqds80ZW1q.gif Edit (3.5 KiB, image/gif; name="CgSLDMLqds80ZW1q.gif")

Download full text (6.6 KiB)

Hi Paride,

Apologies for delayed reply.

We're running 22.04.4 LTS. We switched to Ubuntu from CentOS7 last year. We
installed the *nginx* and *php8.1-fpm* using apt install immediately after
the initial install.

We've been applying regular updates using:

apt update
apt upgrade

We use the Nginx *gzip_static on;* directive to speed up the transfers of
large HTML, Javascript and CSS files. This has been working fine in CentOS7
along with the initial Ubuntu 22.04 installation. However, after a recent
"apt update; apt upgrade" cycle, Nginx no longer sends these files with
*Content-Encoding: gzip* even though nothing else has changed.

We've confirmed that our test browsers are including the *Accept-Encoding:
gzip, deflate, br, zstd* header in their request.

Our Nginx config files are:

*/etc/nginx/nginx.conf*

    user www-data;
    worker_processes auto;
    pid /run/nginx.pid;
    include /etc/nginx/modules-enabled/*.conf;

    events {
         worker_connections 768;
         # multi_accept on;
    }

http {

         ##
         # Basic Settings
         ##

         sendfile on;
         tcp_nopush on;
         types_hash_max_size 2048;
         # server_tokens off;

# server_names_hash_bucket_size 64;
# server_name_in_redirect off;

include /etc/nginx/mime.types;
default_type application/octet-stream;

         ##
         # SSL Settings
         ##

         ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref:
    POODLE
         ssl_prefer_server_ciphers on;

         ##
         # Logging Settings
         ##

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

         ##
         # Gzip Settings
         ##

# gzip on;

         # gzip_vary on;
         # gzip_proxied any;
         # gzip_comp_level 6;
         # gzip_buffers 16 8k;
         # gzip_http_version 1.1;
         # gzip_types text/plain text/css application/json
    application/javascript text/xml application/xml application/xml+rss
    text/javascript;

         ##
         # Virtual Host Configs
         ##

         include /etc/nginx/conf.d/*.conf;
         include /etc/nginx/sites-enabled/*;
    }

*/etc/nginx/sites-available/villadeus.com
*

server {
server_name villadeus.com;

         auth_basic           off;
         root /usr/share/nginx/villadeus.com;
         index index.php index.html index.htm;

         location /.well-known/ {
             root /usr/share/nginx/html;
         }

         location / {
         gzip_static on;
             try_files $uri $uri/index.html =404;
         }

         location ~ \.html$ {
             gzip_static on;
         }
         location ~ \.css$ {
             gzip_static on;
         }
         location ~ \.js$ {
             gzip_static on;
         }

         error_page 404 /nginx/404.html;
         location = /nginx/40x.html {
             log_not_found off;
             access_log off;
         }

         location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
             expires max;
             log_not_found off;
         }

listen 208.99.196.229:443 ssl; # managed by Certbot
...

Hi Paride,

Apologies for delayed reply.

We're running 22.04.4 LTS. We switched to Ubuntu from CentOS7 last year. We 
installed the *nginx* and *php8.1-fpm* using apt install immediately after 
the initial install.

We've been applying regular updates using:

apt update
apt upgrade

We use the Nginx *gzip_static on;* directive to speed up the transfers of 
large HTML, Javascript and CSS files. This has been working fine in CentOS7 
along with the initial Ubuntu 22.04 installation. However, after a recent 
"apt update; apt upgrade" cycle, Nginx no longer sends these files with 
*Content-Encoding: gzip* even though nothing else has changed.

We've confirmed that our test browsers are including the *Accept-Encoding: 
gzip, deflate, br, zstd* header in their request.

Our Nginx config files are:

*/etc/nginx/nginx.conf*

user www-data;
    worker_processes auto;
    pid /run/nginx.pid;
    include /etc/nginx/modules-enabled/*.conf;

events {
         worker_connections 768;
         # multi_accept on;
    }

http {

##
         # Basic Settings
         ##

sendfile on;
         tcp_nopush on;
         types_hash_max_size 2048;
         # server_tokens off;

# server_names_hash_bucket_size 64;
         # server_name_in_redirect off;

include /etc/nginx/mime.types;
         default_type application/octet-stream;

##
         # SSL Settings
         ##

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref:
    POODLE
         ssl_prefer_server_ciphers on;

##
         # Logging Settings
         ##

access_log /var/log/nginx/access.log;
         error_log /var/log/nginx/error.log;

##
         # Gzip Settings
         ##

# gzip on;

# gzip_vary on;
         # gzip_proxied any;
         # gzip_comp_level 6;
         # gzip_buffers 16 8k;
         # gzip_http_version 1.1;
         # gzip_types text/plain text/css application/json
    application/javascript text/xml application/xml application/xml+rss
    text/javascript;

##
         # Virtual Host Configs
         ##

include /etc/nginx/conf.d/*.conf;
         include /etc/nginx/sites-enabled/*;
    }

*/etc/nginx/sites-available/villadeus.com
*

server {
         server_name  villadeus.com;

auth_basic           off;
         root /usr/share/nginx/villadeus.com;
         index index.php index.html index.htm;

location /.well-known/ {
             root /usr/share/nginx/html;
         }

location / {
         gzip_static on;
             try_files $uri $uri/index.html =404;
         }

location ~ \.html$ {
             gzip_static on;
         }
         location ~ \.css$ {
             gzip_static on;
         }
         location ~ \.js$ {
             gzip_static on;
         }

error_page 404 /nginx/404.html;
         location = /nginx/40x.html {
             log_not_found off;
             access_log off;
         }

location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
             expires max;
             log_not_found off;
         }

listen 208.99.196.229:443 ssl; # managed by Certbot
         ssl_certificate /etc/letsencrypt/live/villadeus.com/fullchain.pem;
    # managed by Certbot
         ssl_certificate_key
    /etc/letsencrypt/live/villadeus.com/privkey.pem; # managed by Certbot
         http2 on;
    }
    server {
         location /.well-known/ {
             root /usr/share/nginx/html;
         }

if ($host = villadeus.com) {
             return 301 https://$host$request_uri;
         } # managed by Certbot

listen       208.99.196.229:80;
         server_name  villadeus.com;
         #return 404; # managed by Certbot
    }

*nginx -V*

nginx version: nginx/1.18.0 (Ubuntu)
    built with OpenSSL 3.0.2 15 Mar 2022
    TLS SNI support enabled
    configure arguments: --with-cc-opt='-g -O2
    -ffile-prefix-map=/build/nginx-zctdR4/nginx-1.18.0=. -flto=auto
    -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong
    -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2'
    --with-ld-opt='-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects
    -flto=auto -Wl,-z,relro -Wl,-z,now -fPIC' --prefix=/usr/share/nginx
    --conf-path=/etc/nginx/nginx.conf
    --http-log-path=/var/log/nginx/access.log
    --error-log-path=/var/log/nginx/error.log
    --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid
    --modules-path=/usr/lib/nginx/modules
    --http-client-body-temp-path=/var/lib/nginx/body
    --http-fastcgi-temp-path=/var/lib/nginx/fastcgi
    --http-proxy-temp-path=/var/lib/nginx/proxy
    --http-scgi-temp-path=/var/lib/nginx/scgi
    --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-compat --with-debug
    --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module
    --with-http_realip_module --with-http_auth_request_module
    --with-http_v2_module --with-http_dav_module --with-http_slice_module
    --with-threads
    --add-dynamic-module=/build/nginx-zctdR4/nginx-1.18.0/debian/modules/http-geoip2
    --with-http_addition_module --with-http_gunzip_module
    *--with-http_gzip_static_module* --with-http_sub_module

On one server, we upgraded to the latest available stable Nginx using their 
official repo/process:

https://nginx.org/en/linux_packages.html#Ubuntu

The *gzip_static on;* directive is working properly again on the newly 
installed nginx/1.26.1 service.

From this, we concluded that something recently happened to the 
nginx/1.18.0 provided by the Ubuntu repo that breaks the *gzip_static on;* 
directiv.

Please reach out to me if I can provide additional details.

Thanks much,

Ovid

Pinnion, Inc. 	*Ovid Stavrica*
Chief Technology Officer
Pinnion, Inc.

(800) 818-2039 x806 <tel:8008182039,,806>

On 6/20/24 4:07 AM, Paride Legovini wrote:
> Hello and thanks for your bug report. Looks like you are using
>
>    Package: nginx 1.18.0-6ubuntu14.4
>
> from jammy-updated, but that update is from almost one year ago, and the
> (only) d/changelog entry looks very unrelated to what you are reporting,
> see [1].
>
> Can you please share more information on the update history of your
> servers, and on what update specifically you believe caused the
> regression? Also: could you provide a minimal reproducer? Something on
> these lines:
>
>    lxc launch ubuntu:jammy nginx-j
>    lxc exec nginx-j bash
>    apt install nginx
>    # configure nginx
>    curl [someoptions]http://localhost/...
>    # X should be Y but is Z
>
> Thanks!
>
> [1]https://launchpad.net/ubuntu/+source/nginx/1.18.0-6ubuntu14.4
>
> ** Changed in: nginx (Ubuntu)
>         Status: New => Incomplete
>

Revision history for this message

Simon Déziel (sdeziel) wrote on 2024-06-25:

#4

@pinnion, I just checked a random asset from your site (https://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1) and the `gzip` version has a different ETag than the uncompressed one:

$ wget -qSO /dev/null https://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1 2>&1 | grep ETag
  ETag: "66720e58-15601"
$ wget -qSO /dev/null https://villadeus.com/wp-includes/js/jquery/jquery.min.js.gz?ver=3.7.1 2>&1 | grep ETag
  ETag: "66720f0c-7656"
$ wget --compress gzip -qSO /dev/null https://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1 2>&1 | grep ETag
  ETag: "66720f0c-7656"

On one of my working system with `gzip_static` working, the ETag is always identical which makes sense if I understand the feature properly.

Are you 100% sure your `gzip`'ed files are based on the uncompressed ones? `ls --full-time` might be handy to confirm.

Revision history for this message

Ovid Stavrica (pinnion) wrote on 2024-06-25:

#5

fxnJOZkeehpa3Jrc.gif Edit (3.5 KiB, image/gif; name="fxnJOZkeehpa3Jrc.gif")

We got sloppy there by not following up with `touch *` on the directory
contents. While this indeed breaks cache coherence, it doesn't seem to
affect the gzip_static behavior.

We resolved the problem on https://villadeus.com/ by manually updating to
nginx/1.26.1

The problem is still observable on a staging server that's still running
"nginx/1.18.0 (Ubuntu)" sourced from the Ubuntu repo:
https://staging.dashboard.villadeus.com/js/select2.full.min.js

Pinnion, Inc. *Ovid Stavrica*
Chief Technology Officer
Pinnion, Inc.

(800) 818-2039 x806 <tel:8008182039,,806>

On 6/25/24 8:29 AM, Simon Déziel wrote:
> @pinnion, I just checked a random asset from your site
> (https://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1)
> and the `gzip` version has a different ETag than the uncompressed one:
>
>
> $ wget -qSO /dev/nullhttps://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1 2>&1 | grep ETag
> ETag: "66720e58-15601"
> $ wget -qSO /dev/nullhttps://villadeus.com/wp-includes/js/jquery/jquery.min.js.gz?ver=3.7.1 2>&1 | grep ETag
> ETag: "66720f0c-7656"
> $ wget --compress gzip -qSO /dev/nullhttps://villadeus.com/wp-includes/js/jquery/jquery.min.js?ver=3.7.1 2>&1 | grep ETag
> ETag: "66720f0c-7656"
>
> On one of my working system with `gzip_static` working, the ETag is
> always identical which makes sense if I understand the feature properly.
>
> Are you 100% sure your `gzip`'ed files are based on the uncompressed
> ones? `ls --full-time` might be handy to confirm.
>

Revision history for this message

Simon Déziel (sdeziel) wrote on 2024-06-25:

#6

@pinnion, I'm still seeing differing ETags between those 2:

$ wget -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js.gz 2>&1 | grep ETag
ETag: "667af11b-520f"
$ wget --compress gzip -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js 2>&1 | grep ETag
ETag: "667af11b-124fd"

Also, re `touch *`, I'm not entirely sure on how you are doing the "ahead of time compression" but I get good result from `gzip --best --keep --force --no-name` which cause the `.gz` files to be created with a newer timestamp naturally.

Revision history for this message

Ovid Stavrica (pinnion) wrote on 2024-06-25:

#7

Our script is using:

gzip -c $FILENAME > $FILENAME.gz;
touch $FILENAME.gz -r $FILENAME

FWIW, we also just tried:

rm select2.full.min.js.gz
gzip --best --keep --force --no-name select2.full.min.js

The time stamps between the two files are an exact match:

<snip> ... 75005 2024-06-25 09:32:27.487493248 -0700 select2.full.min.js
<snip> ... 20677 2024-06-25 09:32:27.487493248 -0700 select2.full.min.js.gz

We also restarted the staging server for sanity.

I can confirm that ETag values remain unchanged from your noted values using
your exact wget tests.

Open to suggestions.

On 6/25/24 10:03 AM, Simon Déziel wrote:
> @pinnion, I'm still seeing differing ETags between those 2:
>
> $ wget -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js.gz 2>&1 | grep ETag
> ETag: "667af11b-520f"
> $ wget --compress gzip -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js 2>&1 | grep ETag
> ETag: "667af11b-124fd"
>
>
> Also, re `touch *`, I'm not entirely sure on how you are doing the "ahead of time compression" but I get good result from `gzip --best --keep --force --no-name` which cause the `.gz` files to be created with a newer timestamp naturally.
>

Revision history for this message

Ovid Stavrica (pinnion) wrote on 2024-06-25:

#8

Some, additional thoughts:

We have two separate files:
select2.full.min.js.gz
select2.full.min.js

A discussion from from 2015 suggests that Nginx computes ETags using last
modified time and content length.

SOURCE:
https://serverfault.com/questions/690341/algorithm-behind-nginx-etag-generation

It is possible to configure Nginx with: gzip_static always;

This will always retrieve data from the gzip'ed file and then use the
ngx_http_gunzip_module module to decompress the data before sending it to
clients that do not support "Content-Encoding: gzip"; additional checks may
be warranted to verify that the ETag truly only depends on the file timestamp.

On 6/25/24 10:03 AM, Simon Déziel wrote:
> @pinnion, I'm still seeing differing ETags between those 2:
>
> $ wget -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js.gz 2>&1 | grep ETag
> ETag: "667af11b-520f"
> $ wget --compress gzip -qSO /dev/null https://staging.dashboard.villadeus.com/js/select2.full.min.js 2>&1 | grep ETag
> ETag: "667af11b-124fd"
>
>
> Also, re `touch *`, I'm not entirely sure on how you are doing the "ahead of time compression" but I get good result from `gzip --best --keep --force --no-name` which cause the `.gz` files to be created with a newer timestamp naturally.
>

Revision history for this message

Simon Déziel (sdeziel) wrote on 2024-06-25 (last edit on 2024-06-25):

#9

Maybe your nginx is not using the `.gz` file due to access permission issue or something like that. Anyway, I'd be surprised for it to be a package issue as I have the exact same version as you and have no issue. Feels like an environmental issue to me.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2024-08-25:

#10

[Expired for nginx (Ubuntu) because there has been no activity for 60 days.]

Changed in nginx (Ubuntu):
status:	Incomplete → Expired

Ubuntu
nginx package

gzip_static directive stopped working

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntunginx package

gzip_static directive stopped working

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Ubuntu
nginx package