ordering cycle after reboot

Bug #1930203 reported by reemguin
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
tang (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Fix Released
Medium
Rafael Lopez

Bug Description

[ Impact ]
The enabled tangd.socket service starting on boot is unreliable, due to
the start job being deleted as a result of a systemd ordering cycle.
Users relying on tangd to be started on boot have to manually
check that the service started after a reboot, or implement custom
workarounds to ensure the same.

The fix removes the opportunity for an ordering cycle to occur in the
unit by moving dependencies out of the socket unit and changing the
WantedBy to sockets.target (what it should be for a socket unit).

[ Test Plan ]
The bug is reproduced by installing 'tang' (version 7-1build1) on a machine running Focal 20.04.6 or earlier. The unit should be enabled by default.
 `sudo apt install tang`

After installing the package, ensure the tang service is enabled.
 `systemctl is-enabled tang`

reboot the server, and check that the tangd.socket service is running
 `systemctl status tangd.socket`

The service may or may not be running depending how systemd ordered
the startup jobs for this boot. You can simply repeat rebooting the server
and eventually at some point the service will not come up after a boot.

Regardless if the service is started on boot, you can see similar messages
to these in the system log:
Apr 5 05:01:05 tangtest-vm-2 kernel: systemd[1]: sockets.target: Found ordering cycle on tangd.socket/start
Apr 5 05:01:05 tangtest-vm-2 kernel: systemd[1]: sockets.target: Found dependency on tangd-update.service/start
Apr 5 05:01:05 tangtest-vm-2 kernel: systemd[1]: sockets.target: Found dependency on basic.target/start
Apr 5 05:01:05 tangtest-vm-2 kernel: systemd[1]: sockets.target: Found dependency on sockets.target/start
Apr 5 05:01:05 tangtest-vm-2 kernel: systemd[1]: sockets.target: Job tangd.socket/start deleted to break ordering cycle starting with sockets.target/start

The fix requires modifying the tangd.socket unit WantedBy from multi-user.target to sockets.target. This means the 'enabled' status relies on links in a different directory before and after upgrade. The upgrade ought to remove the old multi-user.target link. So assuming the unit is 'enabled', the only links existing should be:

Before upgrade:
/etc/systemd/system/multi-user.target.wants/tangd.socket

After upgrade:
/etc/systemd/system/sockets.target.wants/tangd.socket

[ Where problems could occur ]
The tang service may not start correctly. If it is made to be part of
a systemd dependency chain with other services, those services may also
be impacted/fail to start.

Since the systemd units are altered, even if the service starts it may
change the way the tang service was originally configured to run. Other
applications/clients relying on the tang service may experiences issues
if the service is not running as originally configured in prior release.

[ Other Info ]
The proposed fix is derived from an upstream fix:
https://github.com/latchset/tang/commit/77785125fb56

There is a minor modification to the diff since the Ubuntu package source file for tangd.socket has a '.in' extension, upstream does not.

Versions starting with v8 already contain the fix:
$ git describe --contains 77785125fb56
v8~9

Since we are updating the WantedBy link from multi-user to socket via a change in postinst, a downgrade using `apt install tang={version}` will not have a postinst to account for this. The best way to downgrade is to remove the package, and then install the desired version to add the correct link.

Original bug text below:
----------
I had the same issue, described here: https://bugzilla.redhat.com/show_bug.cgi?id=1792173
This issue was found, because tangd didn't provide the advertisement payload anymore, after reboot.

Ubuntu version: 20.04.2
Package version of tang: 7-1build1

The bug could be fixed by the recommended changes from Renaud Métrich 2020-01-17 08:35:08 UTC.
----------

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in tang (Ubuntu):
status: New → Confirmed
Revision history for this message
Andy Sayler (andy.sayler) wrote :

I had a similar issue with Tang on Ubuntu 20.04.2 that caused tangd.socket not not auto start after most reboots. Simualr to the RHEL bug linked in th previous comment, I was able to fix it by replacing "WantedBy=multi-user.target" with "WantedBy=sockets.target" in /lib/systemd/system/tangd.socket.

Changed in tang (Ubuntu Focal):
status: New → Confirmed
Revision history for this message
Rafael Lopez (rafael.lopez) wrote :

debdiff for focal

Changed in tang (Ubuntu Focal):
status: Confirmed → In Progress
Changed in tang (Ubuntu):
status: Confirmed → Fix Released
Changed in tang (Ubuntu Focal):
importance: Undecided → Medium
assignee: nobody → Rafael Lopez (rafael.lopez)
tags: added: sts
description: updated
description: updated
tags: added: se-sponsor-halves
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Thanks for the patch, Rafael! I've tested your changes, and confirmed it fixes the issue. The debdiff needed some minor adjustments before uploading, written below for reference:
- added your patch file to the new changelog entry
- run 'update-maintainer' (as this is the first 'ubuntu' revision of the tang package)

Since those were only minor, I went ahead and modified it before sponsoring for the focal upload queue.
Thanks again for the fix!

description: updated
description: updated
Revision history for this message
Robie Basak (racb) wrote :

> +-WantedBy=multi-user.target
> ++WantedBy=sockets.target

The above is in the Install section. I checked what happens on upgrade, and found that /etc/systemd/system/multi-user.target.wants/tangd.socket does not get replaced with /etc/systemd/system/sockets.target.wants/tangd.socket. So this SRU will result in a different configuration depending on whether a user upgrades up to it, or installs from focal-updates directly for the first time.

I assume this is an oversight? If so, please fix.

Revision history for this message
Robie Basak (racb) wrote :

It's probably also worth adding to the Test Plan to ensure the configuration is indeed the same in both cases.

Robie Basak (racb)
Changed in tang (Ubuntu Focal):
status: In Progress → Incomplete
description: updated
Revision history for this message
Rafael Lopez (rafael.lopez) wrote :

Thanks for the feedback Robie. I have updated the SRU template with information regarding the links.

Heitor, I have attached a new debdiff including your changes plus some additional changes to fix the items mentioned by Robie.

Changed in tang (Ubuntu Focal):
status: Incomplete → In Progress
Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Just to circle back on this one, Rafael and I discussed the postinst scripts last week, and we've agreed on some further changes to make it more inline with what's in other packages for similar situations (changing systemd services between pkg versions). We'll have a new one in the bug soon, so I'm removing the current debdiff to prevent an accidental upload.

Revision history for this message
Rafael Lopez (rafael.lopez) wrote :

Adding new debdiff, this one should cover everything we discussed.

Revision history for this message
Heitor Alves de Siqueira (halves) wrote :

Thank you for the new debdiff, Rafael! I've tested this new version on both upgrade path and fresh install, and the service files are cleaned correctly.

Sponsored for focal. Thank you for your contribution!

Revision history for this message
Steve Langasek (vorlon) wrote : Please test proposed package

Hello reemguin, or anyone else affected,

Accepted tang into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/tang/7-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in tang (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Rafael Lopez (rafael.lopez) wrote :
Download full text (8.9 KiB)

Have verified that the -proposed package fixes the bug, details below.

### launch focal vm, install tang
lxc launch ubuntu:focal tangtest --vm
(ssh to vm)
sudo apt update
sudo apt upgrade
sudo apt install tang
systemctl status tangd.socket (active, enabled)

### confirm issue - rebooted vm until tang did not start, check logs for systemd job deleted messages
ubuntu@tangtest:~$ sudo systemctl status tangd.socket
● tangd.socket - Tang Server socket
     Loaded: loaded (/lib/systemd/system/tangd.socket; enabled; vendor preset: enabled)
     Active: inactive (dead)
     Listen: [::]:80 (Stream)
   Accepted: 0; Connected: 0;
ubuntu@tangtest:~$
ubuntu@tangtest:~$ grep 'Job' /var/log/syslog
Jun 5 00:53:24 tangtest kernel: systemd[1]: sockets.target: Job tangd.socket/start deleted to break ordering cycle starting with sockets.target/start
ubuntu@tangtest:~$

### added focal-proposed repository, install proposed tang
ubuntu@tangtest:~$ sudo apt-cache policy tang
tang:
  Installed: 7-1build1
  Candidate: 7-1ubuntu0.1
  Version table:
     7-1ubuntu0.1 500
        500 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 Packages
 *** 7-1build1 500
        500 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages
        100 /var/lib/dpkg/status

### tested upgrade from pre-enabled state (result - unit is active, enabled and symlinks correct)
ubuntu@tangtest:~$ sudo apt install tang
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be upgraded:
  tang
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 15.5 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-proposed/universe amd64 tang amd64 7-1ubuntu0.1 [15.5 kB]
Fetched 15.5 kB in 1s (11.8 kB/s)
(Reading database ... 76087 files and directories currently installed.)
Preparing to unpack .../tang_7-1ubuntu0.1_amd64.deb ...
Unpacking tang (7-1ubuntu0.1) over (7-1build1) ...
Setting up tang (7-1ubuntu0.1) ...
Created symlink /etc/systemd/system/sockets.target.wants/tangd.socket → /lib/systemd/system/tangd.socket.
tangd-keygen.service is a disabled or a static unit not running, not starting it.
tangd-update.service is a disabled or a static unit not running, not starting it.
Processing triggers for man-db (2.9.1-1) ...
ubuntu@tangtest:~$ sudo systemctl status tangd.socket
● tangd.socket - Tang Server socket
     Loaded: loaded (/lib/systemd/system/tangd.socket; enabled; vendor preset: enabled)
     Active: active (listening) since Mon 2023-06-05 00:58:19 UTC; 15s ago
   Triggers: ● tangd@0.service
     Listen: [::]:80 (Stream)
   Accepted: 0; Connected: 0;
      Tasks: 0 (limit: 1153)
     Memory: 0B
     CGroup: /system.slice/tangd.socket

Jun 05 00:58:19 tangtest systemd[1]: Listening on Tang Server socket.
ubuntu@tangtest:~$ ls -l /etc/systemd/system/multi-user.target.wants/tangd.socket
ls: cannot access '/etc/systemd/system/multi-user.target.wants/tangd.socket': No such file or directory
ubuntu@tangtest:~$ ls -l /etc/systemd/system/sockets.target.wants/tangd.socket
lrwxrwxrwx 1 root ro...

Read more...

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Robie Basak (racb) wrote : Update Released

The verification of the Stable Release Update for tang has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package tang - 7-1ubuntu0.1

---------------
tang (7-1ubuntu0.1) focal; urgency=medium

  * Fix systemd service files to avoid ordering cycle (LP: #1930203)
    - d/p/lp-1930203-Fixed-Ordering-cycle-found-causing-spurious-issues-dur.patch
  * d/control: update-maintainer
  * d/tang.postinst: clean up multi-user.target.wants link from previous version

 -- Rafael Lopez <email address hidden> Fri, 05 May 2023 09:02:31 +0000

Changed in tang (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.