Activity log for bug #1964881

Date Who What changed Old value New value Message
2022-03-15 06:06:04 Evren Yurtesen bug added bug
2022-03-17 11:12:09 Evren Yurtesen description In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` The reason of missing `catalina.out` in Ubuntu 20.04 is that the `/var/log/tomcat` does not have group write permission. Therefore `syslog` user which executes `rsyslogd` is not able to write to `/var/log/tomcat` folder. A common problem between Ubuntu 20.04/22.04 is that the `/etc/rsyslog.d/tomcat9.conf` file does not have the following setting: `fileOwner="tomcat"` (which exists in upstream Debian `tomcat9` package supplied `/etc/rsyslog.d/tomcat9.conf` file) Because of the missing `fileOwner` setting, the `catalina.out` is created with `syslog:adm` ownerships. However the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done.
2022-03-18 10:51:02 Evren Yurtesen tags focal jammy apport-collected focal jammy
2022-03-18 10:51:03 Evren Yurtesen description In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-03-18 10:51:03 Evren Yurtesen attachment added Dependencies.txt https://bugs.launchpad.net/bugs/1964881/+attachment/5570365/+files/Dependencies.txt
2022-03-18 10:51:04 Evren Yurtesen attachment added ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1964881/+attachment/5570366/+files/ProcCpuinfoMinimal.txt
2022-03-18 10:51:05 Evren Yurtesen attachment added ProcEnviron.txt https://bugs.launchpad.net/bugs/1964881/+attachment/5570367/+files/ProcEnviron.txt
2022-03-28 13:34:12 Evren Yurtesen attachment added Fixes LP: #1964881 https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1964881/+attachment/5573761/+files/1-1.0-1ubuntu1.debdiff
2022-03-28 13:34:35 Evren Yurtesen bug added subscriber Ubuntu Sponsors Team
2022-03-29 08:29:21 Evren Yurtesen attachment added Fixes LP: #1964881 and #1861881 on 20.04/focal https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1964881/+attachment/5574178/+files/1-1.0-1ubuntu1.debdiff
2022-03-30 13:20:58 Evren Yurtesen bug watch added https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008668
2022-04-04 10:18:26 Simon Chopin nominated for series Ubuntu Focal
2022-04-04 10:18:26 Simon Chopin bug task added tomcat9 (Ubuntu Focal)
2022-04-04 10:18:26 Simon Chopin nominated for series Ubuntu Jammy
2022-04-04 10:18:26 Simon Chopin bug task added tomcat9 (Ubuntu Jammy)
2022-04-27 15:21:25 Robie Basak tags apport-collected focal jammy apport-collected focal jammy server-todo
2022-05-10 06:04:33 Launchpad Janitor tomcat9 (Ubuntu): status New Confirmed
2022-05-10 06:04:33 Launchpad Janitor tomcat9 (Ubuntu Focal): status New Confirmed
2022-05-10 06:04:33 Launchpad Janitor tomcat9 (Ubuntu Jammy): status New Confirmed
2022-05-10 06:04:37 Sebastian bug added subscriber Sebastian
2022-06-08 14:50:26 Andreas Hasenack tomcat9 (Ubuntu): assignee Andreas Hasenack (ahasenack)
2022-06-08 14:50:53 Andreas Hasenack bug added subscriber Ubuntu Server
2022-06-08 14:51:01 Andreas Hasenack bug added subscriber Canonical Server Team
2022-06-15 15:03:25 Christian Ehrhardt  tomcat9 (Ubuntu Focal): assignee Andreas Hasenack (ahasenack)
2022-06-15 15:03:29 Christian Ehrhardt  tomcat9 (Ubuntu Jammy): assignee Andreas Hasenack (ahasenack)
2022-06-23 19:07:30 Andreas Hasenack tomcat9 (Ubuntu): status Confirmed In Progress
2022-06-23 21:17:10 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340
2022-06-27 16:56:28 Andreas Hasenack tomcat9 (Ubuntu Jammy): status Confirmed In Progress
2022-06-27 16:56:36 Andreas Hasenack tomcat9 (Ubuntu Jammy): status In Progress Confirmed
2022-06-27 21:25:07 Launchpad Janitor tomcat9 (Ubuntu): status In Progress Fix Released
2022-07-08 04:57:14 Mathew Hodson bug task added tomcat9 (Debian)
2022-07-09 02:04:21 Bug Watch Updater tomcat9 (Debian): status Unknown New
2022-07-19 20:41:11 Andreas Hasenack tomcat9 (Ubuntu Jammy): status Confirmed In Progress
2022-07-19 20:41:13 Andreas Hasenack tomcat9 (Ubuntu Focal): status Confirmed In Progress
2022-07-20 17:28:08 Andreas Hasenack description In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Plan] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. * if other testing is appropriate to perform before landing this update, this should also be described here. [Where problems could occur] * Think about what the upload changes in the software. Imagine the change is wrong or breaks something else: how would this show up? * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This must '''never''' be "None" or "Low", or entirely an argument as to why your upload is low risk. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-20 19:11:54 Andreas Hasenack description [Impact] * An explanation of the effects of the bug on users and * justification for backporting the fix to the stable release. * In addition, it is helpful, but not required, to include an explanation of how the upload fixes this bug. [Test Plan] * detailed instructions how to reproduce the bug * these should allow someone who is not familiar with the affected package to reproduce the bug and verify that the updated package fixes the problem. * if other testing is appropriate to perform before landing this update, this should also be described here. [Where problems could occur] * Think about what the upload changes in the software. Imagine the change is wrong or breaks something else: how would this show up? * It is assumed that any SRU candidate patch is well-tested before upload and has a low overall risk of regression, but it's important to make the effort to think about what ''could'' happen in the event of a regression. * This must '''never''' be "None" or "Low", or entirely an argument as to why your upload is low risk. * This both shows the SRU team that the risks have been considered, and provides guidance to testers in regression-testing the SRU. [Other Info] * Anything else you think is useful to include * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact]  * An explanation of the effects of the bug on users and  * justification for backporting the fix to the stable release.  * In addition, it is helpful, but not required, to include an    explanation of how the upload fixes this bug. [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur]  * Think about what the upload changes in the software. Imagine the change is    wrong or breaks something else: how would this show up?  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This must '''never''' be "None" or "Low", or entirely an argument as to why    your upload is low risk.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-20 19:17:43 Andreas Hasenack description [Impact]  * An explanation of the effects of the bug on users and  * justification for backporting the fix to the stable release.  * In addition, it is helpful, but not required, to include an    explanation of how the upload fixes this bug. [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur]  * Think about what the upload changes in the software. Imagine the change is    wrong or breaks something else: how would this show up?  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This must '''never''' be "None" or "Low", or entirely an argument as to why    your upload is low risk.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur]  * Think about what the upload changes in the software. Imagine the change is    wrong or breaks something else: how would this show up?  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This must '''never''' be "None" or "Low", or entirely an argument as to why    your upload is low risk.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-20 19:28:47 Andreas Hasenack description [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur]  * Think about what the upload changes in the software. Imagine the change is    wrong or breaks something else: how would this show up?  * It is assumed that any SRU candidate patch is well-tested before    upload and has a low overall risk of regression, but it's important    to make the effort to think about what ''could'' happen in the    event of a regression.  * This must '''never''' be "None" or "Low", or entirely an argument as to why    your upload is low risk.  * This both shows the SRU team that the risks have been considered,    and provides guidance to testers in regression-testing the SRU. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-20 19:33:52 Andreas Hasenack description [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info]  * Anything else you think is useful to include  * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board  * and address these questions in advance [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info] Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340 [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-21 13:31:29 Andreas Hasenack description [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep catalina\\.out /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info] Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340 [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info] Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340 [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-21 13:46:15 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/427217
2022-07-21 13:53:47 Launchpad Janitor merge proposal linked https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/427218
2022-07-21 13:54:53 Andreas Hasenack description [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info] Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340 [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True [Impact] Log handling in tomcat9 is broken in several ways: a) logrotate fails to rotate the catalina.out log file b) rsyslog is configured to chown the catalina.out log file to the tomcat user, but lacks the privileges to do so (in Ubuntu, rsyslog runs unprivileged) c) even though on a fresh install tomcat9 is able to log to /var/log/tomcat9/catalina.out via rsyslog, a simple upgrade or reinstall of the tomcat9 package will break that logging by changing the ownership of catalina.out to the "tomcat9" user, in which case rsyslog won't be able to write to it anymore (as soon as it closes the fd and tries to reopen it) [Test Plan] Create a container or VM for the ubuntu release under test. Here we will use lxc, and the commands and outputs below will be shown for jammy: lxc launch ubuntu:jammy j-tomcat9-logging lxc shell j-tomcat9-logging apt update && apt install tomcat9 Observe that the /var/log/tomcat9 directory has permissions 02770 and that the catalina.out file in it is owned by syslog:adm: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9 total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 314 Jul 20 18:32 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt But here the problems start, and these are the ones fixed by this SRU: a) rsyslog is complaining that it can't change the ownership of catalina.out: root@j-tomcat9-logging:~# grep -E "chown.*catalina\.out.*not permitted" /var/log/syslog Jul 20 18:32:22 j-tomcat9-logging rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] b) logrotate fails: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf error: error opening /var/log/tomcat9/catalina.out: Permission denied And catalina.out remains unrotated: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 12 drwxrws--- 1 tomcat adm 188 Jul 20 18:32 . drwxrwxr-x 1 root syslog 430 Jul 20 18:33 .. -rw-r----- 1 tomcat adm 5994 Jul 20 18:32 catalina.2022-07-20.log -rw-r----- 1 syslog adm 3522 Jul 20 18:32 catalina.out -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) if the package is reinstalled, or an update without this fix becomes available and is applied, the catalina.out file will have incorrect ownership and rsyslog won't be able to write to it anymore: before reinstall: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 3523 Jul 20 18:49 catalina.out after reinstall: root@j-tomcat9-logging:~# apt install --reinstall tomcat9 -y Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 tomcat adm 3797 Jul 20 18:49 catalina.out And logging is broken: root@j-tomcat9-logging:~# grep -E "catalina\.out.*Permission denied" /var/log/syslog Jul 20 18:49:59 j-tomcat9-logging rsyslogd: file '/var/log/tomcat9/catalina.out': open error: Permission denied [v8.2112.0 try https://www.rsyslog.com/e/2433 ] Now install the tomcat9 package from proposed. a) rsyslog won't complain anymore about failing to open or chown the file: root@j-tomcat9-logging:~# systemctl stop rsyslog.service syslog.socket root@j-tomcat9-logging:~# > /var/log/syslog root@j-tomcat9-logging:~# systemctl start rsyslog.service syslog.socket root@j-tomcat9-logging:~# grep rsyslogd /var/log/syslog Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3) from systemd. [v8.2112.0] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: imklog: cannot open kernel log (/proc/kmsg): Permission denied. Jul 20 18:55:09 j-tomcat9-logging rsyslogd: activation of module imklog failed [v8.2112.0 try https://www.rsyslog.com/e/2145 ] Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's groupid changed to 111 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: rsyslogd's userid changed to 104 Jul 20 18:55:09 j-tomcat9-logging rsyslogd: [origin software="rsyslogd" swVersion="8.2112.0" x-pid="5590" x-info="https://www.rsyslog.com"] start b) This time logrotate works, and the catalina.out file will be rotated: root@j-tomcat9-logging:~# logrotate -f /etc/logrotate.conf root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/ total 24 drwxrws--- 1 tomcat adm 216 Jul 20 18:39 . drwxrwxr-x 1 root syslog 612 Jul 20 18:39 .. -rw-r----- 1 tomcat adm 12487 Jul 20 18:37 catalina.2022-07-20.log -rw-r----- 1 syslog adm 0 Jul 20 18:39 catalina.out -rw-r----- 1 syslog adm 7699 Jul 20 18:39 catalina.out.1 -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost.2022-07-20.log -rw-r----- 1 tomcat adm 0 Jul 20 18:32 localhost_access_log.2022-07-20.txt c) reinstalling the package won't break logging again: root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 7974 Jul 20 19:10 /var/log/tomcat9/catalina.out root@j-tomcat9-logging:~# apt install tomcat9 -y --reinstall Reading package lists... Done (...) Processing triggers for rsyslog (8.2112.0-2ubuntu2.2) ... root@j-tomcat9-logging:~# ls -la /var/log/tomcat9/catalina.out -rw-r----- 1 syslog adm 12152 Jul 20 19:11 /var/log/tomcat9/catalina.out [Where problems could occur] These logging problems have been ongoing for quite some time, at least since Focal (20.04), so it's quite possible that users have made local configuration changes to avoid it. Part of the fix in this SRU is in the tomcat9.postinst maintainer script, which is difficult for local users to override, so it's possible that this update will undo, or conflict, with whatever local fixes were made. It's hard to predict what it could be, and trying to be smart about it carries its own set of risks and complexities. I didn't go down that road, trying to keep the change simple and easy to understand. [Other Info] Older logging bug: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 MP proposing this fix for Kinetic, with some discussion and considerations: https://code.launchpad.net/~ahasenack/ubuntu/+source/tomcat9/+git/tomcat9/+merge/425340 For focal, this SRU is also changing the permissions of /var/log/tomcat9 from 02750 to 02770. In jammy and later, it's 02770 already. [Original Description] In Ubuntu 20.04, with `tomcat9-9.0.31-1ubuntu0.1` (latest) package, `logrotated` is not able to write to `/var/log/tomcat/catalina.out` This could be fixed in a newer package but was not backported: https://bugs.launchpad.net/ubuntu/+source/tomcat9/+bug/1861881 In Ubuntu 22.04, with `tomcat9-9.0.58-1` (latest) package, `logrotated` is not able to rotate `/var/log/tomcat/catalina.out` Because the `catalina.out` is created with `syslog:adm` ownerships. `syslog` user does not have enough permissions to change this. This causes following error: rsyslogd: error during config processing: omfile: chown for file '/var/log/tomcat9/catalina.out' failed: Operation not permitted [v8.2112.0 try https://www.rsyslog.com/e/2207 ] At the same time, the `/etc/logrotate.d/tomcat9` has `su tomcat adm` directive. Therefore the `logrotated` is not able to truncate the `/var/log/tomcat/catalina.out` This causes logrotate to copy the contents of `/var/log/tomcat/catalina.out` to as if it would be rotated. As `catalina.out` is never truncated, each rotated file ends up having the contents of `catalina.out` from the beginning of the tomcat installation. This causes the log sizes to keep increasing as no actual log rotation is being done. --- ProblemType: Bug ApportVersion: 2.20.11-0ubuntu79 Architecture: amd64 CasperMD5CheckResult: pass DistroRelease: Ubuntu 22.04 InstallationDate: Installed on 2022-02-27 (18 days ago) InstallationMedia: Ubuntu 22.04 LTS "Jammy Jellyfish" - Alpha amd64 (20220121) Package: tomcat9 9.0.58-1 PackageArchitecture: all ProcVersionSignature: Ubuntu 5.15.0-18.18-generic 5.15.12 RebootRequiredPkgs: Error: path contained symlinks. Tags: jammy Uname: Linux 5.15.0-18-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True
2022-07-26 11:00:34 Robie Basak tomcat9 (Ubuntu Jammy): status In Progress Fix Committed
2022-07-26 11:00:37 Robie Basak bug added subscriber Ubuntu Stable Release Updates Team
2022-07-26 11:00:39 Robie Basak bug added subscriber SRU Verification
2022-07-26 11:00:41 Robie Basak tags apport-collected focal jammy server-todo apport-collected focal jammy server-todo verification-needed verification-needed-jammy
2022-07-26 11:01:02 Robie Basak removed subscriber Ubuntu Sponsors Team
2022-07-26 11:01:17 Robie Basak tomcat9 (Ubuntu Focal): status In Progress Fix Committed
2022-07-26 11:01:22 Robie Basak tags apport-collected focal jammy server-todo verification-needed verification-needed-jammy apport-collected focal jammy server-todo verification-needed verification-needed-focal verification-needed-jammy
2022-07-27 14:02:20 Andreas Hasenack tags apport-collected focal jammy server-todo verification-needed verification-needed-focal verification-needed-jammy apport-collected focal jammy server-todo verification-done-jammy verification-needed verification-needed-focal
2022-07-27 14:18:05 Andreas Hasenack tags apport-collected focal jammy server-todo verification-done-jammy verification-needed verification-needed-focal apport-collected focal jammy server-todo verification-done-focal verification-done-jammy verification-needed
2022-08-03 12:33:26 Andreas Hasenack tags apport-collected focal jammy server-todo verification-done-focal verification-done-jammy verification-needed apport-collected focal jammy server-todo verification-done verification-done-focal verification-done-jammy
2022-08-11 15:44:01 Launchpad Janitor tomcat9 (Ubuntu Jammy): status Fix Committed Fix Released
2022-08-11 15:44:08 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2022-08-16 22:59:23 Launchpad Janitor tomcat9 (Ubuntu Focal): status Fix Committed Fix Released
2023-03-01 17:46:13 Bryce Harrington removed subscriber Canonical Server
2023-08-01 19:11:28 Andreas Hasenack bug task added tomcat10 (Ubuntu)
2023-08-01 19:11:58 Andreas Hasenack bug task deleted tomcat10 (Ubuntu Focal)
2023-08-01 19:12:02 Andreas Hasenack bug task deleted tomcat10 (Ubuntu Jammy)
2023-08-01 19:21:01 Andreas Hasenack bug added subscriber Andreas Hasenack
2023-08-01 20:13:34 Ubuntu Foundations Team Bug Bot tags apport-collected focal jammy server-todo verification-done verification-done-focal verification-done-jammy apport-collected focal jammy patch server-todo verification-done verification-done-focal verification-done-jammy
2023-08-01 20:13:39 Ubuntu Foundations Team Bug Bot bug added subscriber Ubuntu Sponsors
2023-08-02 15:17:23 Robie Basak tags apport-collected focal jammy patch server-todo verification-done verification-done-focal verification-done-jammy apport-collected focal jammy patch verification-done verification-done-focal verification-done-jammy
2023-08-03 09:56:54 Julian Andres Klode removed subscriber Ubuntu Sponsors