Activity log for bug #1957320

Date Who What changed Old value New value Message
2022-01-12 19:11:10 Athos Ribeiro bug added bug
2022-03-15 10:47:18 Athos Ribeiro bug added subscriber Ubuntu Server
2022-03-15 10:47:27 Athos Ribeiro nginx (Ubuntu): importance Undecided Low
2022-09-15 09:46:41 Robie Basak nominated for series Ubuntu Jammy
2022-09-15 09:46:41 Robie Basak bug task added nginx (Ubuntu Jammy)
2022-09-15 09:46:48 Robie Basak nginx (Ubuntu): status New Fix Released
2022-09-15 09:46:51 Robie Basak nginx (Ubuntu Jammy): importance Undecided Low
2023-03-15 21:23:42 Athos Ribeiro nginx (Ubuntu Jammy): status New Triaged
2023-05-15 12:29:03 Athos Ribeiro tags bitesize
2023-05-29 11:21:28 Michał Małoszewski nginx (Ubuntu Jammy): assignee Michał Małoszewski (michal-maloszewski99)
2023-05-30 17:15:03 Michał Małoszewski nginx (Ubuntu Jammy): status Triaged In Progress
2023-05-30 17:49:16 Launchpad Janitor merge proposal linked https://code.launchpad.net/~michal-maloszewski99/ubuntu/+source/nginx/+git/nginx/+merge/443827
2023-05-31 14:58:01 Michał Małoszewski description As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server { listen unix:/run/serve-files.socket; root /var/www/files; location / { try_files $uri =404; } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; gzip_disable "msie6"; ## # Virtual Host Configs ## include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. Example of successful output: No errors in the error.log file. [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-05-31 15:06:30 Michał Małoszewski tags bitesize bitesize server-todo
2023-05-31 19:07:29 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http { ## # Basic Settings ## sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; ## # Logging Settings ## access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; ## # Gzip Settings ## gzip on; gzip_disable "msie6"; ## # Virtual Host Configs ## include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. Example of successful output: No errors in the error.log file. [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. Moreover: × nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /run/systemd/system/service.d └─zzz-lxc-service.conf Active: failed (Result: exit-code) since Wed 2023-05-31 15:35:38 UTC; 46s ago Docs: man:nginx(8) Process: 896 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE) Example of successful output: No errors in the error.log file. nginx can be restarted [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-01 10:41:37 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. Moreover: × nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /run/systemd/system/service.d └─zzz-lxc-service.conf Active: failed (Result: exit-code) since Wed 2023-05-31 15:35:38 UTC; 46s ago Docs: man:nginx(8) Process: 896 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE) Example of successful output: No errors in the error.log file. nginx can be restarted [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: × nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /run/systemd/system/service.d └─zzz-lxc-service.conf Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago Docs: man:nginx(8) Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. nginx can be restarted [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-01 10:42:37 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: × nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Drop-In: /run/systemd/system/service.d └─zzz-lxc-service.conf Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago Docs: man:nginx(8) Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. nginx can be restarted [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-01 10:48:44 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * TBD ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * TBD [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-01 11:36:44 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * TBD [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be rebuilding nginx against newer versions of its build dependencies. [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-02 09:11:57 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Open the /etc/nginx/nginx.conf file. The content of that file should match: http {     ##     # Basic Settings     ##     sendfile on;     tcp_nopush on;     tcp_nodelay on;     keepalive_timeout 65;     types_hash_max_size 2048;     include /etc/nginx/mime.types;     default_type application/octet-stream;     ##     # Logging Settings     ##     access_log /var/log/nginx/access.log;     error_log /var/log/nginx/error.log;     ##     # Gzip Settings     ##     gzip on;     gzip_disable "msie6";     ##     # Virtual Host Configs     ## # include /etc/nginx/conf.d/*.conf;     include /etc/nginx/sites-enabled/*; } Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be rebuilding nginx against newer versions of its build dependencies. [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-02 11:35:56 Michał Małoszewski tags bitesize server-todo server-todo
2023-06-13 09:34:46 Christian Ehrhardt  description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. [Other Info] * It might be a problem with reproduce it if you use: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF In that case please copy that part of the code and paste it into the /etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. [Other Info] * none ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-13 17:27:45 Michał Małoszewski description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. [Other Info] * none ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. * After installing nginx, causing the issue and then installing nginx with fix or simply re-installing the same version - the nginx fails to upgrade, because the former issue that is caused makes the service non-startable and the upgrade will trigger a service restart in postinst. Since that fails it cannot upgrade. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-15 08:31:56 Christian Ehrhardt  description [Impact] * Restarting nginx fails on Jammy. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. * After installing nginx, causing the issue and then installing nginx with fix or simply re-installing the same version - the nginx fails to upgrade, because the former issue that is caused makes the service non-startable and the upgrade will trigger a service restart in postinst. Since that fails it cannot upgrade. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. * After installing nginx, causing the issue and then installing nginx with   fix or simply re-installing the same version - the nginx fails to upgrade,   because the former issue that is caused makes the service non-startable and   the upgrade will trigger a service restart in postinst. Since that fails it   cannot upgrade. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-15 08:41:11 Christian Ehrhardt  description [Impact] * Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * A future possible code refactoring can affect the functionality of   ngx_close_listening_sockets, which can introduce a regression in that specific fix. * Moreover another possible source of regressions is the fact that we will be   rebuilding nginx against newer versions of its build dependencies. * After installing nginx, causing the issue and then installing nginx with   fix or simply re-installing the same version - the nginx fails to upgrade,   because the former issue that is caused makes the service non-startable and   the upgrade will trigger a service restart in postinst. Since that fails it   cannot upgrade. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b [Impact] * Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log. * That issue is caused by the behavior of the custom socket closing code in   ngx_master_process_cycle(). * The fix is to replace the custom socket closing code in   ngx_master_process_cycle() by the ngx_close_listening_sockets() call. [Test Plan] Make a container for testing: $ lxc launch ubuntu-daily:jammy jammy-test $ lxc shell jammy-test Type in: # apt install -y nginx # systemctl stop nginx Then type in: # cat << EOF > /etc/nginx/sites-enabled/serve-files server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files \$uri =404;     } } EOF # service nginx start # service nginx stop # service nginx restart Enter the /var/log/nginx/error.log Example of failed output: You will be able to see the error logs in the error.log file. 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error) 2023/06/01 10:39:02 [emerg] 895#895: still could not bind() Moreover: systemctl status nginx × nginx.service - A high performance web server and a reverse proxy server      Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)     Drop-In: /run/systemd/system/service.d              └─zzz-lxc-service.conf      Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago        Docs: man:nginx(8)     Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)     Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE) Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server... Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error) Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind() Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'. Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server. Example of successful output: No errors in the error.log file. Nginx can be restarted without any problems. [Where problems could occur] * After installing nginx, causing the issue and then installing nginx with   fix or simply re-installing the same version - the nginx fails to upgrade,   because the former issue that is caused makes the service non-startable and   the upgrade will trigger a service restart in postinst. Since that fails it   cannot upgrade. If you are affected, just remove the offending socket and restart the service - after that (due to the fix) you will not be affected by this again. ---------------------------original bug report-------------------------------------------- As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process. This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4]. To reproduce the issue, run the following commands from a jammy machine: # apt install -y nginx # systemctl stop nginx # mkdir -p /var/www/files # echo hello > /var/www/files/hello # cat << EOF > /run/serve-files.socket server {     listen unix:/run/serve-files.socket;     root /var/www/files;     location / {         try_files $uri =404;     } } EOF # systemctl start nginx Verify it works with: # echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket And restart the service: # systemctl restart nginx This will throw an error and the service will end in a failed state. Verify that the socket file in /run/serve-files.socket was not removed. Removing the socket file should allow you to restart the service. [1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965 [2] https://trac.nginx.org/nginx/ticket/753 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111 [4] http://hg.nginx.org/nginx/rev/7cbf6389194b
2023-06-28 07:36:46 Chris Halse Rogers nginx (Ubuntu Jammy): status In Progress Fix Committed
2023-06-28 07:36:47 Chris Halse Rogers bug added subscriber Ubuntu Stable Release Updates Team
2023-06-28 07:36:51 Chris Halse Rogers bug added subscriber SRU Verification
2023-06-28 07:36:56 Chris Halse Rogers tags server-todo server-todo verification-needed verification-needed-jammy
2023-06-28 17:17:54 Michał Małoszewski tags server-todo verification-needed verification-needed-jammy server-todo verification-done verification-done-jammy
2023-07-10 11:14:51 Launchpad Janitor nginx (Ubuntu Jammy): status Fix Committed Fix Released
2023-07-10 11:15:16 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team