| 2022-01-12 19:11:10 |
Athos Ribeiro |
bug |
|
|
added bug |
| 2022-03-15 10:47:18 |
Athos Ribeiro |
bug |
|
|
added subscriber Ubuntu Server |
| 2022-03-15 10:47:27 |
Athos Ribeiro |
nginx (Ubuntu): importance |
Undecided |
Low |
|
| 2022-09-15 09:46:41 |
Robie Basak |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-09-15 09:46:41 |
Robie Basak |
bug task added |
|
nginx (Ubuntu Jammy) |
|
| 2022-09-15 09:46:48 |
Robie Basak |
nginx (Ubuntu): status |
New |
Fix Released |
|
| 2022-09-15 09:46:51 |
Robie Basak |
nginx (Ubuntu Jammy): importance |
Undecided |
Low |
|
| 2023-03-15 21:23:42 |
Athos Ribeiro |
nginx (Ubuntu Jammy): status |
New |
Triaged |
|
| 2023-05-15 12:29:03 |
Athos Ribeiro |
tags |
|
bitesize |
|
| 2023-05-29 11:21:28 |
Michał Małoszewski |
nginx (Ubuntu Jammy): assignee |
|
Michał Małoszewski (michal-maloszewski99) |
|
| 2023-05-30 17:15:03 |
Michał Małoszewski |
nginx (Ubuntu Jammy): status |
Triaged |
In Progress |
|
| 2023-05-30 17:49:16 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~michal-maloszewski99/ubuntu/+source/nginx/+git/nginx/+merge/443827 |
|
| 2023-05-31 14:58:01 |
Michał Małoszewski |
description |
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
Example of successful output:
No errors in the error.log file.
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-05-31 15:06:30 |
Michał Małoszewski |
tags |
bitesize |
bitesize server-todo |
|
| 2023-05-31 19:07:29 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
Example of successful output:
No errors in the error.log file.
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
Moreover:
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Wed 2023-05-31 15:35:38 UTC; 46s ago
Docs: man:nginx(8)
Process: 896 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Example of successful output:
No errors in the error.log file.
nginx can be restarted
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-01 10:41:37 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
Moreover:
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Wed 2023-05-31 15:35:38 UTC; 46s ago
Docs: man:nginx(8)
Process: 896 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Example of successful output:
No errors in the error.log file.
nginx can be restarted
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover:
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
nginx can be restarted
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-01 10:42:37 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover:
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
nginx can be restarted
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-01 10:48:44 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* TBD
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* TBD
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-01 11:36:44 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* TBD
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-02 09:11:57 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Open the /etc/nginx/nginx.conf file.
The content of that file should match:
http {
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
gzip_disable "msie6";
##
# Virtual Host Configs
##
# include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-02 11:35:56 |
Michał Małoszewski |
tags |
bitesize server-todo |
server-todo |
|
| 2023-06-13 09:34:46 |
Christian Ehrhardt |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* It might be a problem with reproduce it if you use:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
In that case please copy that part of the code and paste it into the
/etc/nginx/sites-enabled/serve-files which you have to create yourself because it does not exist. I do not know why using that command is a problem, but it happened to me.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* none
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-13 17:27:45 |
Michał Małoszewski |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
[Other Info]
* none
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
* After installing nginx, causing the issue and then installing nginx with
fix or simply re-installing the same version - the nginx fails to upgrade,
because the former issue that is caused makes the service non-startable and
the upgrade will trigger a service restart in postinst. Since that fails it
cannot upgrade.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-15 08:31:56 |
Christian Ehrhardt |
description |
[Impact]
* Restarting nginx fails on Jammy. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
* After installing nginx, causing the issue and then installing nginx with
fix or simply re-installing the same version - the nginx fails to upgrade,
because the former issue that is caused makes the service non-startable and
the upgrade will trigger a service restart in postinst. Since that fails it
cannot upgrade.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
* After installing nginx, causing the issue and then installing nginx with
fix or simply re-installing the same version - the nginx fails to upgrade,
because the former issue that is caused makes the service non-startable and
the upgrade will trigger a service restart in postinst. Since that fails it
cannot upgrade.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-15 08:41:11 |
Christian Ehrhardt |
description |
[Impact]
* Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* A future possible code refactoring can affect the functionality of
ngx_close_listening_sockets, which can introduce a regression in that specific fix.
* Moreover another possible source of regressions is the fact that we will be
rebuilding nginx against newer versions of its build dependencies.
* After installing nginx, causing the issue and then installing nginx with
fix or simply re-installing the same version - the nginx fails to upgrade,
because the former issue that is caused makes the service non-startable and
the upgrade will trigger a service restart in postinst. Since that fails it
cannot upgrade.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
[Impact]
* Restarting nginx fails on Jammy is listening on unix sockets is configured. It produces the errors in the error.log.
* That issue is caused by the behavior of the custom socket closing code in
ngx_master_process_cycle().
* The fix is to replace the custom socket closing code in
ngx_master_process_cycle() by the ngx_close_listening_sockets() call.
[Test Plan]
Make a container for testing:
$ lxc launch ubuntu-daily:jammy jammy-test
$ lxc shell jammy-test
Type in:
# apt install -y nginx
# systemctl stop nginx
Then type in:
# cat << EOF > /etc/nginx/sites-enabled/serve-files
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files \$uri =404;
}
}
EOF
# service nginx start
# service nginx stop
# service nginx restart
Enter the /var/log/nginx/error.log
Example of failed output:
You will be able to see the error logs in the error.log file.
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: bind() to unix:/run/serve-files.socket failed (98: Unknown error)
2023/06/01 10:39:02 [emerg] 895#895: still could not bind()
Moreover: systemctl status nginx
× nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Drop-In: /run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: failed (Result: exit-code) since Thu 2023-06-01 10:39:05 UTC; 56s ago
Docs: man:nginx(8)
Process: 894 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 895 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
Jun 01 10:39:02 jammy-test systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 01 10:39:02 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:03 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:04 jammy-test nginx[895]: nginx: [emerg] bind() to unix:/run/serve-files.socket failed (98: Unknown error)
Jun 01 10:39:05 jammy-test nginx[895]: nginx: [emerg] still could not bind()
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 01 10:39:05 jammy-test systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 01 10:39:05 jammy-test systemd[1]: Failed to start A high performance web server and a reverse proxy server.
Example of successful output:
No errors in the error.log file.
Nginx can be restarted without any problems.
[Where problems could occur]
* After installing nginx, causing the issue and then installing nginx with
fix or simply re-installing the same version - the nginx fails to upgrade,
because the former issue that is caused makes the service non-startable and
the upgrade will trigger a service restart in postinst. Since that fails it
cannot upgrade.
If you are affected, just remove the offending socket and restart the
service - after that (due to the fix) you will not be affected by this
again.
---------------------------original bug report--------------------------------------------
As discussed in [1], there is a bug in nginx [2] which makes Ubuntu's systemd unit restarts fail when nginx is litening on a unix domain socket. This happens because nginx fails to remove the socket during its shutdown process.
This issue has been reported in Debian in [3], and has been fixed upstream since 1.19.1, with the following patch: [4].
To reproduce the issue, run the following commands from a jammy machine:
# apt install -y nginx
# systemctl stop nginx
# mkdir -p /var/www/files
# echo hello > /var/www/files/hello
# cat << EOF > /run/serve-files.socket
server {
listen unix:/run/serve-files.socket;
root /var/www/files;
location / {
try_files $uri =404;
}
}
EOF
# systemctl start nginx
Verify it works with:
# echo -e "GET /hello HTTP/1.0\r\n" | netcat -U /run/serve-files.socket
And restart the service:
# systemctl restart nginx
This will throw an error and the service will end in a failed state.
Verify that the socket file in /run/serve-files.socket was not removed.
Removing the socket file should allow you to restart the service.
[1] https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1919965
[2] https://trac.nginx.org/nginx/ticket/753
[3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=821111
[4] http://hg.nginx.org/nginx/rev/7cbf6389194b |
|
| 2023-06-28 07:36:46 |
Chris Halse Rogers |
nginx (Ubuntu Jammy): status |
In Progress |
Fix Committed |
|
| 2023-06-28 07:36:47 |
Chris Halse Rogers |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-06-28 07:36:51 |
Chris Halse Rogers |
bug |
|
|
added subscriber SRU Verification |
| 2023-06-28 07:36:56 |
Chris Halse Rogers |
tags |
server-todo |
server-todo verification-needed verification-needed-jammy |
|
| 2023-06-28 17:17:54 |
Michał Małoszewski |
tags |
server-todo verification-needed verification-needed-jammy |
server-todo verification-done verification-done-jammy |
|
| 2023-07-10 11:14:51 |
Launchpad Janitor |
nginx (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2023-07-10 11:15:16 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
