"reload" action breaks further actions

Bug #1721635 reported by Andreas Hasenack
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
lighttpd (Debian)
Fix Released
Unknown
lighttpd (Ubuntu)
Fix Released
High
Andreas Hasenack

Bug Description

After you issue a "service lighttpd reload" (or call /etc/init.d/lighttpd directly with the same action), all further actions stop working.

Example:

root@xenial-lighttpd-reload-1707312:~# service lighttpd start

root@xenial-lighttpd-reload-1707312:~# ps fxaw
  PID TTY STAT TIME COMMAND
(...)
 4361 ? Ss 0:00 /usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf

So pid is 4361, and things are good. "status" is happy:
root@xenial-lighttpd-reload-1707312:~# service lighttpd status
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-10-05 20:08:43 UTC; 45s ago
  Process: 4356 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 4361 (lighttpd)

"restart" works and we get a new pid:

root@xenial-lighttpd-reload-1707312:~# service lighttpd restart
root@xenial-lighttpd-reload-1707312:~# pidof lighttpd
5281
root@xenial-lighttpd-reload-1707312:~# service lighttpd status
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2017-10-05 20:09:58 UTC; 6s ago
  Process: 5276 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 5281 (lighttpd)

Now let's "reload". The systemd job has no reload target, so the reload code from the sysv script is used:

root@xenial-lighttpd-reload-1707312:~# service lighttpd reload
 * Reloading web server configuration lighttpd
   ...done.
root@xenial-lighttpd-reload-1707312:~# pidof lighttpd
5912

We got a new PID, because that reload is more like a restart. But now things are broken:
root@xenial-lighttpd-reload-1707312:~# service lighttpd status
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2017-10-05 20:10:49 UTC; 1min 2s ago
  Process: 5281 ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
  Process: 5276 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 5281 (code=exited, status=0/SUCCESS)

And "restart" fails as well, because it tries to kill the process it thinks should be there, fails, and also fails to start a new one up because the port is already in use:

root@xenial-lighttpd-reload-1707312:~# service lighttpd status
● lighttpd.service - Lighttpd Daemon
   Loaded: loaded (/lib/systemd/system/lighttpd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2017-10-05 20:12:07 UTC; 1s ago
  Process: 6853 ExecStart=/usr/sbin/lighttpd -D -f /etc/lighttpd/lighttpd.conf (code=exited, status=255)
  Process: 6848 ExecStartPre=/usr/sbin/lighttpd -t -f /etc/lighttpd/lighttpd.conf (code=exited, status=0/SUCCESS)
 Main PID: 6853 (code=exited, status=255)

Oct 05 20:12:07 xenial-lighttpd-reload-1707312 systemd[1]: Starting Lighttpd Daemon...
Oct 05 20:12:07 xenial-lighttpd-reload-1707312 lighttpd[6848]: Syntax OK
Oct 05 20:12:07 xenial-lighttpd-reload-1707312 systemd[1]: Started Lighttpd Daemon.
Oct 05 20:12:07 xenial-lighttpd-reload-1707312 lighttpd[6853]: 2017-10-05 20:12:07: (network.c.409) can't bind to port: 80 Address already in use

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: lighttpd 1.4.35-4ubuntu2
ProcVersionSignature: Ubuntu 4.13.0-12.13-generic 4.13.3
Uname: Linux 4.13.0-12-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
Date: Thu Oct 5 20:06:12 2017
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: lighttpd
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Revision history for this message
Vasya Pupkin (shadowlmd) wrote :

Hello,

This patch for lighttpd.service file changes default signal systemd sends to lighttpd when stopping service. It also sets a 60 seconds timeout after that SIGKILL will be sent to lighttpd and it's children if they survive first SIGINT. This makes `systemctl force-reload lighttpd.service` do a graceful restart, so using /etc/init.d/lighttpd is not even needed.

It's also a good idea to remove "reload" action from /etc/init.d/lighttpd completely because it's not supposed to be there, lighttpd doesn't support it.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lighttpd.service.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Changed in lighttpd (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
status: New → In Progress
importance: Undecided → High
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Test packages are building here for xenial, zesty and artful: https://launchpad.net/~ahasenack/+archive/ubuntu/lighttpd-reload-1707312-1721635

What they do:
a) remove the "reload" action from the SysV initscript
b) apply the systemd service file changes suggested here

Expected outcome:
- slight change in behavior for all actions that involve a stop: the command will block until all existing connections terminate, or the timeout (60s) passes. During that time, new connections won't be accepted.
- no more reload action
- force-reload is an alias to restart in systemd, so the force-reload action from the SysV initscript isn't run, but its code was fixed to behave as a graceful restart anyway
- all actions continue to work after force-reload or restart as expected

Of all these, the one that I wonder about for an SRU is the first one. It's a change in behavior. Now, a restart (for example) can take up to 60s, whereas before it was "immediate". Unfortunately there is no way that I can see where we could add a new graceful-restart action. If we add it to just the SysV initscript, we run into this bug again.

I'm not too worried about removing the "reload" action because it was a) broken; b) incorrect, as it was restarting the service instead of reloading. There is no proper reload in this version of lighttpd: is going to be supported in the upcoming version 1.4.46 via a USR1 signal (https://git.lighttpd.net/lighttpd/lighttpd1.4.git/commit/?h=0ae6bab4a97f12a0c93200df36ac1741696eeed5)

Changed in lighttpd (Debian):
status: Unknown → New
Changed in lighttpd (Debian):
status: New → Fix Released
Revision history for this message
gstrauss (gstrauss) wrote :

lighttpd 1.4.46 was released in Oct 2017, almost 3 years ago.
Is there any reason this is still open?

Revision history for this message
Vasya Pupkin (shadowlmd) wrote :

Fix discussed here never arrived into Ubuntu 16.04.

Revision history for this message
gstrauss (gstrauss) wrote :

I am a lighttpd developer. Most of the issues I see here can, if needed, have patches backported by me in 5-10 mins.

However, it would be useful if someone with Ubuntu packaging privileges helps shepherd changes.

Revision history for this message
Vasya Pupkin (shadowlmd) wrote :

The easiest fix for Ubuntu 16.04 (and probably 18.04 too) is:

$ sudo mkdir -p /etc/systemd/system/lighttpd.service.d
$ echo '[Service]
KillMode=mixed
KillSignal=SIGINT
TimeoutStopSec=60' | sudo tee /etc/systemd/system/lighttpd.service.d/sigint.conf
$ sudo systemctl daemon-reload

Ubuntu 20.04 already contains a new Lighttpd release with configuration reload support via SIGUSR1 signal.

Revision history for this message
gstrauss (gstrauss) wrote :

Yes, the fix is trivial. Someone needs to do it in those versions.

Changed in lighttpd (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
gstrauss (gstrauss) wrote :

lighttpd 1.4.59 is part of Ubuntu 21.04 (Hirsute Hippo)

no reason to keep this open for years if the Ubuntu lighttpd package maintainers are never going to get around to backporting the changes

Changed in lighttpd (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.