systemd locks up due to incorrect handling of time zone changes

[Matt Coleman]

[Impact]

This bug can cause systemd timers to lock up systems using the Europe/Dublin timezone if certain OnCalendar values are used. If the timer is enabled at boot, this bug prevents the system from booting. If the timer is started later, systemd becomes unresponsive.

[Test Plan]

* Create a systemd service and corresponding timer scheduled for 01:00 on Sundays:

$ cat /etc/systemd/system/lp-1966800-reproducer.timer
 [Unit]
 Description=LP 1966800 reproducer

 [Timer]
 OnCalendar=Sun *-*-* 01:00:00
 Persistent=true

 [Install]
 WantedBy=timers.target

 $ cat /etc/systemd/system/lp-1966800-reproducer.service
 [Service]
 ExecStart=/usr/bin/echo "LP 1966800 reproducer"

 $ systemctl daemon-reload

* Set the date to 21 March, 2021 in the Europe/Dublin timezone (need to disable NTP first):

 $ timedatectl set-ntp false
 $ timedatectl set-time "2021-03-21"
 $ timedatectl set-timezone Europe/Dublin

* Try to start the timer, and observe that systemd becomes unresponsive. This command will time out, as will `systemctl status` etc.

 $ systemctl start lp-1966800-reproducer.timer

[Where problems could occur]

The patch makes changes in the calendar spec code, which is used by systemd timers to determine the next time to trigger a timer. If this patch breaks something, it would be related to the triggering of systemd timers.

[Original Description]

Recently on systems in Ireland, systemd became unresponsive due the change from GMT to Irish Standard Time. This is due to Ireland being unique in having their standard time during the summer, unlike most regions.

Related to: https://bugzilla.redhat.com/show_bug.cgi?id=1941335
Fixed by: https://github.com/systemd/systemd-stable/commit/a8b66ca9af811148b67ee952ab32748f88b8bba3

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

This is fixed in the systemd present in Ubuntu 21.10 and later

[Lukas Märdian (slyon)]

A fix for this issue has been staged in git for the next Focal SRU: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594

Thank you @enr0n for providing the merge proposal!

[Łukasz Zemczak (sil2100)]

Hello Matt, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.18) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-gke-5.15/5.15.0-1015.18~20.04.1 (arm64, amd64)
netplan.io/0.104-0ubuntu2~20.04.2 (armhf)
linux-hwe-5.15/5.15.0-48.54~20.04.1 (armhf, amd64)
gvfs/1.44.1-1ubuntu1.1 (ppc64el)
linux-oracle-5.15/5.15.0-1016.20~20.04.1 (arm64)
linux-oracle-5.13/5.13.0-1036.43~20.04.1 (arm64)
linux-hwe-5.11/5.11.0-61.61 (arm64)
systemd/245.4-4ubuntu3.18 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Nick Rosbrook (enr0n)]

The autopkgtest regressions were resolved by retries.

I have verified this fix on focal using the test plan above:

nr@clean-focal-amd64:~$ apt-cache policy systemd
systemd:
  Installed: 245.4-4ubuntu3.18
  Candidate: 245.4-4ubuntu3.18
  Version table:
 *** 245.4-4ubuntu3.18 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     245.4-4ubuntu3.17 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     245.4-4ubuntu3.15 500
        500 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages
     245.4-4ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

nr@clean-focal-amd64:~$ cat /etc/systemd/system/lp-1966800-reproducer.timer
[Unit]
Description=LP 1966800 reproducer

[Timer]
OnCalendar=Sun *-*-* 01:00:00
Persistent=true

[Install]
WantedBy=timers.target

nr@clean-focal-amd64:~$ cat /etc/systemd/system/lp-1966800-reproducer.service
[Service]
ExecStart=/usr/bin/echo "LP 1966800 reproducer"

nr@clean-focal-amd64:~$ sudo systemctl daemon-reload
nr@clean-focal-amd64:~$ sudo timedatectl set-ntp false
nr@clean-focal-amd64:~$ sudo timedatectl set-time "2021-03-21"
nr@clean-focal-amd64:~$ sudo timedatectl set-timezone Europe/Dublin
nr@clean-focal-amd64:~$ sudo systemctl start lp-1966800-reproducer.timer
nr@clean-focal-amd64:~$ sudo systemctl status
● clean-focal-amd64
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2022-09-08 18:08:29 IST; 1 years 5 months left
   CGroup: /
           ├─user.slice
           │ └─user-1000.slice
           │ ├─[e-mail] …
           │ │ ├─gsd-xsettings.service
           │ │ │ └─1565 /usr/libexec/gsd-xsettings
           │ │ ├─gvfs-goa-volume-monitor.service
           │ │ │ └─1216 /usr/libexec/gvfs-goa-volume-monitor
           │ │ ├─gsd-power.service
           │ │ │ └─1531 /usr/libexec/gsd-power
           │ │ ├─xdg-permission-store.service
           │ │ │ └─1444 /usr/libexec/xdg-permission-store
           │ │ ├─gsd-sound.service
           │ │ │ └─1552 /usr/libexec/gsd-sound
           │ │ ├─gsd-rfkill.service
           │ │ │ └─1535 /usr/libexec/gsd-rfkill
           │ │ ├─gsd-usb-protection.service
           │ │ │ └─1555 /usr/libexec/gsd-usb-protection
           │ │ ├─gsd-print-notifications.service
           │ │ │ ├─1534 /usr/libexec/gsd-print-notifications
           │ │ │ └─1609 /usr/libexec/gsd-printer
           │ │ ├─evolution-calendar-factory.service
           │ │ │ └─1465 /usr/libexec/evolution-calendar-factory
           │ │ ├─gsd-a11y-settings.service
           │ │ │ └─1517 /usr/libexec/gsd-a11y-settings
           │ │ ├─gsd-wwan.service
           │ │ │ └─1563 /usr/libexec/gsd-wwan
           │ │ ├─pulseaudio.service
           │ │ │ └─1162 /usr/bin/pulseaudio --daemonize=no --log-target=journal
           │ │ ├─gsd-screensaver-proxy.service
           │ │ │ └─1537 /usr/libexec/gsd-screensaver-proxy
           │ │ ├─gsd-media-keys.service

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.18

---------------
systemd (245.4-4ubuntu3.18) focal; urgency=medium

  [ Nick Rosbrook ]
  * core: make sure we don't get confused when setting TERM for a tty fd
    (LP: #1959475)
    File: debian/patches/lp1959475-core-make-sure-we-don-t-get-confused-when-setting-TERM-fo.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b10c6853050dde26665caf3b15444d768d2bc498
  * shared/calendarspec: when mktime() moves us backwards, jump forward
    (LP: #1966800)
    File: debian/patches/lp1966800-shared-calendarspec-when-mktime-moves-us-backwards-jump-f.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594
  * network: do not remove localhost address (LP: #1979951)
    File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=87f872b8c5451f353601fb606e7fd7a479217cef
  * units: remove the restart limit on the modprobe@.service (LP: #1982462)
    File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=68353ffaf3539e6a58ef62a8b50850f56eae29ea

  [ Mustafa Kemal Gilor ]
  * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after
    modules are loaded. Thanks to Alexander Graf <email address hidden>.
    (LP: #1978079)
    Author: Mustafa Kemal Gilor
    File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e60756f2079d6408abdb967127a1d9b9a0eba8c

 -- Nick Rosbrook <email address hidden> Wed, 31 Aug 2022 11:27:33 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.