[SRU] debhelper support override from /etc/tmpfiles.d for systemd

Bug #1748147 reported by Nick Groenen on 2018-02-08
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
debhelper
Fix Released
Unknown
debhelper (Ubuntu)
High
Seyeong Kim
Xenial
Medium
Unassigned
Artful
Medium
Unassigned
Bionic
Medium
Unassigned
rsyslog (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Artful
Undecided
Unassigned
Bionic
Undecided
Unassigned
systemd (Ubuntu)
Undecided
Dimitri John Ledkov
Xenial
Undecided
Unassigned
Artful
Undecided
Unassigned
Bionic
Undecided
Unassigned

Bug Description

[Impact]

/var/log's Permission is going back to 755
after upgrading systemd
if there are rsyslog's configuration on /var/lib/tmpfiles.d/

Affected X, A, B, C

This is because rsyslog's pkg has 00rsyslog.conf and copied it on /var/lib/tmpfiles.d/ when it is installing.
after upgrading systemd, systemd only refresh it's own tmpfiles so disappear conf for 00rsyslog.conf ( it doesn't remove file itself )
so, systemd-tmpfiles --create /var/lib/tmpfiles.d/00rsyslog.conf back permission to 775

[Test Case]

1. deploy 16.04 vm
2. check ll /var (775)
3. apt install --reinstall systemd
4. check ll /var (755)

[Regression Potential]
This fix changes debhelper's override process by using absolute path to filename. so if the other pkgs using debhelper e.g systemd are there, It should be re-build with new debhelper after patching in theory, now only systemd is affected. but building is not affected. also, pkg like rsyslog which is using systemd's tmpfile system need to be changed to use /etc/tmpfiles.d/[SAME_FILENAME_IN_VAR_LIB_TMPFILES.D_FOR_OVERRIDING] instead of 00rsyslog.conf.

[Others]

For this issue, need to fix below pkgs

debhelper
systemd ( rebuilding with new debhelper is needed )
rsyslog ( 00rsyslog.conf to var.conf and location should be /etc/tmpfiles.d, to support override supported by debhelper )

[Original description]

Upgrading or reinstalling the systemd package when using rsyslogd results in bad permissions (0755 instead of 0775) being set on /var/log/. As a consequence of this, rsyslogd can no longer create new files within this directory, resulting in lost log messages.

The default configuration of rsyslogd provided by Ubuntu runs the daemon as syslog:syslog and sets ownership of /var/log to syslog:adm with mode 0775.

Systemd's default tmpfiles configuration sets /var/log to 0755 in /usr/lib/tmpfiles.d/var.conf, however this is overridden in /usr/lib/tmpfiles.d/00rsyslog.conf which is provided by package rsyslog.

It looks as though an upgrade of the systemd package fails to take /usr/lib/tmpfiles.d/00rsyslog.conf into account, as demonstrated below. This results in /var/log receiving mode 0755 instead of the expected 0775:

nick @ log2.be1.ams1:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

nick @ log2.be1.ams1:~ $ apt policy systemd
systemd:
  Installed: 229-4ubuntu21.1
  Candidate: 229-4ubuntu21.1
  Version table:
 *** 229-4ubuntu21.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

nick @ log2.be1.ams1:~ $ apt policy rsyslog
rsyslog:
  Installed: 8.16.0-1ubuntu3
  Candidate: 8.16.0-1ubuntu3
  Version table:
 *** 8.16.0-1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
        100 /var/lib/dpkg/status

nick @ log2.be1.ams1:~ $ grep -F /var/log /usr/lib/tmpfiles.d/var.conf
d /var/log 0755 - - -
f /var/log/wtmp 0664 root utmp -
f /var/log/btmp 0600 root utmp -

nick @ log2.be1.ams1:~ $ cat /usr/lib/tmpfiles.d/00rsyslog.conf
# Override systemd's default tmpfiles.d/var.conf to make /var/log writable by
# the syslog group, so that rsyslog can run as user.
# See tmpfiles.d(5) for details.

# Type Path Mode UID GID Age Argument
d /var/log 0775 root syslog -

nick @ log2.be1.ams1:~ $ ls -ld /var/log
drwxrwxr-x 8 root syslog 4096 Feb 7 13:45 /var/log

nick @ log2.be1.ams1:~ $ sudo apt install --reinstall systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 7 not upgraded.
Need to get 3,634 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 systemd amd64 229-4ubuntu21.1 [3,634 kB]
Fetched 3,634 kB in 0s (24.3 MB/s)
(Reading database ... 86614 files and directories currently installed.)
Preparing to unpack .../systemd_229-4ubuntu21.1_amd64.deb ...
Unpacking systemd (229-4ubuntu21.1) over (229-4ubuntu21.1) ...
Processing triggers for dbus (1.10.6-1ubuntu3.3) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up systemd (229-4ubuntu21.1) ...
addgroup: The group `systemd-journal' already exists as a system group. Exiting.

nick @ log2.be1.ams1:~ $ ls -ld /var/log
drwxr-xr-x 8 root syslog 4096 Feb 7 13:45 /var/log

Nick Groenen (zonii) wrote :

Related/similar issues: #1428540, #1687015

Seyeong Kim (xtrusia) on 2018-03-31
affects: systemd → debhelper
Seyeong Kim (xtrusia) on 2018-03-31
tags: added: sts
Changed in debhelper:
status: Unknown → New
Seyeong Kim (xtrusia) on 2018-04-23
no longer affects: systemd (Ubuntu)
no longer affects: rsyslog (Ubuntu)
tags: added: xenial
Changed in debhelper (Ubuntu):
importance: Undecided → High
Changed in debhelper:
status: New → Fix Committed
Seyeong Kim (xtrusia) on 2018-05-21
description: updated
Seyeong Kim (xtrusia) on 2018-05-21
description: updated
Changed in debhelper:
status: Fix Committed → Fix Released
Seyeong Kim (xtrusia) wrote :
tags: added: sts-sru-needed
Changed in debhelper (Ubuntu):
assignee: nobody → Seyeong Kim (xtrusia)
description: updated
Seyeong Kim (xtrusia) wrote :
Seyeong Kim (xtrusia) wrote :
Seyeong Kim (xtrusia) wrote :
Seyeong Kim (xtrusia) on 2018-05-25
summary: - Upgrading systemd sets incorrect permissions on /var/log/
+ [SRU] debhelper support override from /etc/tmpfiles.d for systemd
Seyeong Kim (xtrusia) on 2018-05-25
description: updated

The attachment "lp1748147_xenial.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Eric Desrochers (slashd) on 2018-05-29
Changed in debhelper (Ubuntu):
status: New → In Progress
Eric Desrochers (slashd) on 2018-05-29
Changed in debhelper (Ubuntu Xenial):
status: New → In Progress
Changed in debhelper (Ubuntu Artful):
status: New → In Progress
Changed in debhelper (Ubuntu Bionic):
status: New → In Progress
Changed in debhelper (Ubuntu Xenial):
assignee: nobody → Seyeong Kim (xtrusia)
Changed in debhelper (Ubuntu Artful):
assignee: nobody → Seyeong Kim (xtrusia)
Changed in debhelper (Ubuntu Bionic):
assignee: nobody → Seyeong Kim (xtrusia)
Changed in debhelper (Ubuntu Xenial):
importance: Undecided → Medium
Changed in debhelper (Ubuntu Artful):
importance: Undecided → Medium
Changed in debhelper (Ubuntu Bionic):
importance: Undecided → Medium
Eric Desrochers (slashd) wrote :

Sponsored in devel release "cosmic"

Eric Desrochers (slashd) on 2018-05-29
Changed in debhelper (Ubuntu Bionic):
status: In Progress → Fix Committed
status: Fix Committed → In Progress
Changed in debhelper (Ubuntu):
status: In Progress → Fix Committed
Seyeong Kim (xtrusia) wrote :
Seyeong Kim (xtrusia) wrote :
Seyeong Kim (xtrusia) wrote :
Eric Desrochers (slashd) wrote :

== Cosmic : excuse page regression ==

# From excuses... page

* autopkgtest for autopkgtest/5.3.1: amd64: Pass, arm64: Pass, armhf: Pass, i386: Regression ♻ , ppc64el: Pass, s390x: Pass[1]

There was 2 regression for autopkgtest, I was able to make the amd64 pass after 4 attempts. After 5 attempts for i386, it still fails.
The failure as nothing to do with the uploaded patch. It's seems to be a network glitch during the autopkgtest. I guess it will succeed eventually at restarting the test over and over again just like it did for 'amd64'.

* autopkgtest for dahdi-linux/1:2.11.1~dfsg-1ubuntu4: amd64: Pass, arm64: Always failed, armhf: Pass, i386: Regression ♻ , ppc64el: Always failed, s390x: Ignored failure[2]

Other architecture (arm64,ppc64el) set to 'Always failed' fails the exact same way. Last one it succeeded was with kernel 4.15.0-20[3]

[1] buildlogs
Network lxdbr0 created
Storage pool default created
Device root added to default
Creating autopkgtest-prepare-3fF
Error: Failed container creation: Get https://images.linuxcontainers.org/streams/v1/index.json: Unable to connect to: images.linuxcontainers.org:443

Error: not found

[2] buildlogs
Building for 4.15.0-22-generic
Building for architecture i686
Building initial module for 4.15.0-22-generic
Error! Build of dahdi_vpmadt032_loader.ko failed for: 4.15.0-22-generic (i686)
Consult the make.log in the build directory

[3] - https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-cosmic/cosmic/i386/d/dahdi-linux/20180502_155702_8fd1e@/log.gz

With that being said, I think it is safe to release debhelper for cosmic to make it a 'Valid Candidate'.

Eric Desrochers (slashd) wrote :

I'll contact the release team.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package debhelper - 11.2.1ubuntu2

---------------
debhelper (11.2.1ubuntu2) cosmic; urgency=medium

  * [d24b1734] Support overrides in tmpfiles.d (LP: #1748147)
    - dh_installsystemd: Use the basename of the "tmpfiles" config
      files. This makes "systemd-tmpfiles --create" search for it
      in both /usr/lib/tmpfiles.d and in /etc/tmpfiles.d. With
      this change the system administrator can now override the
      "tmpfiles" config shipped by the package in
      /usr/libtmpfiles.d.
    - dh_installinit: Ditto.

 -- Seyeong Kim <email address hidden> Thu, 24 May 2018 22:19:34 -0700

Changed in debhelper (Ubuntu):
status: Fix Committed → Fix Released
Eric Desrochers (slashd) wrote :

After a few retried attempts, all the regressions passed, making the new debhelper in Cosmic a "Valid candidate" now.

# excuses... page
debhelper (11.2.1ubuntu1 to 11.2.1ubuntu2)
Maintainer: Ubuntu Developers
5 days old
....
Valid candidate

Eric Desrochers (slashd) wrote :

Sponsored for B,A,X

Łukasz Zemczak (sil2100) wrote :

I'm a bit confused with the description of the bug. The description mentions requiring a rebuild of any affected package, like systemd, for the fix to work - do I understand this correctly? How many packages will need to be rebuilt? Since you mention the need of changing paths, is there risk that after this lands some packages stop building correctly without modification of their source?

Seyeong Kim (xtrusia) on 2018-06-05
description: updated
Seyeong Kim (xtrusia) wrote :

Hello sil2100.

Sorry for making confusion.
I describe it in detail.

This patch changes tmpfile path from absolute path to filename(only).
so, they now support override feature if we put tmpfile conf to /etc/tmpfiles.d/

but in this situation, we need to set same filename as /var/lib/tmpfiles.d/[something] for using override.

For this override feature, debhelper need to be patched. and systemd need to be rebuilt.

but even if systemd is not rebuit, it is working find as like before.

For rsyslogd ( and pkg like this ), it copies 00rsyslogd.conf file to /var/lib/tmpfiles.d/ directly. it works until systemd restarting. when restarting systemd, it recall tmpfiles conf files only systemd has, so in this time 00rsyslogd.conf is ignored. Then /var/log's permission is changed back to 755 (need 775)

so rsyslogd's 00rsyslogd.conf filename need to be changed to var.conf and target should be /etc/tmpfiles.d/ instead of /var/lib/tmpfiles.d/

even if rsyslogd(or pkg like this ) is not patched, it works with current issue ( as this LP ).

Please let me know if you have anything.

I'll update description based on this comment if you are fine with this.

Thanks

Robie Basak (racb) wrote :

> so rsyslogd's 00rsyslogd.conf filename need to be changed to var.conf and target should be /etc/tmpfiles.d/ instead of /var/lib/tmpfiles.d/

This doesn't seem right to me. The point of the use of /etc/tmpfiles.d/ is for local sysadmin override, not an override from a different package. Having different packages use different directories will just lead to a mess.

I'm also doubtful about changing debhelper's behaviour in a stable release when this isn't really a bug in debhelper in the first place. Can this be fixed differently in the stable releases - by adjusting maintainer scripts more directly, for example?

Adam Conrad (adconrad) wrote :

Yeah, this is not remotely a sane solution for this bug. The real bug appears to be that we're not processing all the tmpfiles.d snippets on upgrade. Having one package override another by using /etc is Very Wrong. The bug here is certainly still in debhelper, but it's that we think that running "systemd-tmpfiles /path/to/config.conf" is a useful thing to do. The only correct way to process systemd-tmpfiles configs is as a whole, because order matters. First always wins (hence why 00rsyslogd.conf comes before vars.conf), and this works correctly on boot.

This debconf snippet should really just be re-running systemd-tmpfiles without arguments when installing packages that install tmpfiles.d snippets, probably.

Adam Conrad (adconrad) wrote :

Also, in a discussion with xnox, this turns out to perhaps also be a systemd bug in that it shouldn't be touching that at all.

Dimitri John Ledkov (xnox) wrote :

There are multiple bugs here.

I do not believe testcase of the rsyslog <-> systemd is wrong, and whilst debhelper support is good, is not what would fix rsyslog <-> systemd in Ubuntu.

Changed in systemd (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
status: New → Confirmed
Seyeong Kim (xtrusia) wrote :

@racb, adconrad

Yes, exactly...

As you can see debian bugs.

In the beginning, I meant to fix it like that
e.g running systemd-tmpfiles for all files in /var/lib/tmpfiles.d/ not just for systemd defined files, by running it without argument

But the maintainer didn't accept mine, you can check debian discussion.

I think we can fix this separately from debian?
If we can, i can upload that patch. I tested it as well. and it worked.

Robie Basak (racb) wrote :

I didn't see this as a bug in debhelper because, based on the Debian bug, it seems to me that the use case of one package's tmpfiles.d/ file overriding another was not considered, even if systemd can handle it underneath. From that perspective, debhelper supporting lexical sort based overriding is a feature request in debhelper, not a bug; and rsyslog packaging relying on an unimplemented debhlper "feature" is the bug.

Seyeong Kim (xtrusia) wrote :

FYI

systemd-tmpfiles command running when installing or upgrading systemd

is in debhelper pkg

cat autoscripts/postinst-init-tmpfiles
if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ]; then
 # In case this system is running systemd, we need to ensure that all
 # necessary tmpfiles (if any) are created before starting.
 if [ -d /run/systemd/system ] ; then
  systemd-tmpfiles --create #TMPFILES# >/dev/null || true
 fi
fi

#TMPFILES# has list of tmpfiles conf file owned by systemd only.

Brian Murray (brian-murray) wrote :

I've rejected the uploads in -proposed to declutter the queues and save some other SRU team member's time.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in rsyslog (Ubuntu Artful):
status: New → Confirmed
Changed in rsyslog (Ubuntu Bionic):
status: New → Confirmed
Changed in rsyslog (Ubuntu Xenial):
status: New → Confirmed
Changed in rsyslog (Ubuntu):
status: New → Confirmed
Changed in systemd (Ubuntu Artful):
status: New → Confirmed
Changed in systemd (Ubuntu Bionic):
status: New → Confirmed
Changed in systemd (Ubuntu Xenial):
status: New → Confirmed

An upload of debhelper to artful-proposed has been rejected from the upload queue for the following reason: "I'm rejecting this as their is some ongoing discussion and not every SRU member should have to read the whole bug.".

Dimitri John Ledkov (xnox) wrote :

Proposed fix in systemd. Run systemd-tmpfiles, during postinst, the way it would be run on boot, such that all base files are correct, including any overrides shipped by any other package; systemd; in transient runtime dir.

At the same time, the dh_installinit is silenced to not produce the systemd-tmpfiles snippet which this package does not need.

This solves the issue of integration with rsyslog; generically; without requiring to backport debhelper, nor change rsyslog package.

Changed in rsyslog (Ubuntu):
status: Confirmed → Invalid
Changed in rsyslog (Ubuntu Xenial):
status: Confirmed → Invalid
Changed in rsyslog (Ubuntu Artful):
status: Confirmed → Invalid
Changed in rsyslog (Ubuntu Bionic):
status: Confirmed → Invalid
Changed in systemd (Ubuntu):
status: Confirmed → Fix Committed
Changed in debhelper (Ubuntu Bionic):
assignee: Seyeong Kim (xtrusia) → nobody
status: In Progress → Won't Fix
Changed in debhelper (Ubuntu Artful):
assignee: Seyeong Kim (xtrusia) → nobody
status: In Progress → Won't Fix
Changed in debhelper (Ubuntu Xenial):
assignee: Seyeong Kim (xtrusia) → nobody
status: In Progress → Won't Fix

Hello Nick, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in systemd (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-bionic
Sebastien Bacher (seb128) wrote :

looks like that has been uploaded? unsubscribing the sponsors

I will be testing the updated systemd and will update the bug here once validated.

Tested, works.

Repro (Xenial):
# dpkg -l | grep systemd
ii systemd 229-4ubuntu21.2 amd64 system and service manager

/var# ll
drwxrwxr-x 8 root syslog 4096 Jul 9 06:25 log/ <--775

# apt install --reinstall systemd

/var# ll
drwxr-xr-x 8 root syslog 4096 Jul 9 06:25 log/ <-- 755

Bionic (Verified):
# dpkg -l | grep systemd
ii systemd 237-3ubuntu10.2 amd64 system and service manager

/var# ll
drwxrwxr-x 8 root syslog 4096 Jul 9 13:09 log/ <-- 775

# apt install --reinstall systemd
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 2895 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 systemd amd64 237-3ubuntu10.2 [2895 kB]
...

/var# ll
drwxrwxr-x 8 root syslog 4096 Jul 9 13:09 log/ <-- 775

tags: added: verification-done-bionic
removed: verification-needed-bionic

Waiting on Xenial update for this...

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 237-3ubuntu10.2

---------------
systemd (237-3ubuntu10.2) bionic; urgency=medium

  * logind: backport v238/v239 fixes for handling DRM devices.
    These changes introduce all the fixes that correct handling of open fd's
    related to the DRM devices, as used by for example NVIDIA GPUs. This backport
    includes some refactoring, corrections, and comment updates. This to insure
    that correct history is preserved, code comments match reality, and to ease
    backporting logind fixes in the future SRUs. (LP: #1777099)
  * Disable dh_installinit generation of tmpfiles for the systemd package.
    Replace with a manual safe call to systemd-tmpfiles which will process any
    updates to the tmpfiles shipped by systemd package, taking into account any
    overrides shipped by other packages, sysadmin, or specified in the runtime
    directories. (LP: #1748147)

systemd (237-3ubuntu10.1) bionic; urgency=medium

  [ Dimitri John Ledkov ]
  * hwdb: Fix wlan/rfkill keycode on Dell systems. (LP: #1762385)
  * Cherrypick upstream fix for corrected detection of Virtualbox & Xen.
    (LP: #1768104)
  * Further improve captive portal workarounds.
    Retry any NXDOMAIN results with lower feature levels, instead of just those
    with 'secure' in the domain name. (LP: #1766969)

  [ Michael Biebl ]
  * Add dependencies of libsystemd-shared to Pre-Depends.
    This is necessary so systemctl is functional at all times during a
    dist-upgrade. (Closes: #897986) (LP: #1771791)

  [ Mario Limonciello ]
  * Fix hibernate disk offsets.
    Configure resume offset via sysfs, to enable resume from a swapfile.
    (LP: #1760106)

 -- Dimitri John Ledkov 🌈 <email address hidden> Fri, 22 Jun 2018 13:55:09 +0100

Changed in systemd (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.