Activity log for bug #1651518

Date Who What changed Old value New value Message
2016-12-20 16:49:32 bugproxy bug added bug
2016-12-20 16:49:35 bugproxy tags architecture-ppc64le bugnameltc-150030 severity-high targetmilestone-inin16042
2016-12-20 16:50:03 bugproxy attachment added sosreport from system yellowbee https://bugs.launchpad.net/bugs/1651518/+attachment/4794381/+files/sosreport-yellowbee.150030-20161219162806.tar.bz2
2016-12-20 20:29:32 Ubuntu Foundations Team Bug Bot tags architecture-ppc64le bugnameltc-150030 severity-high targetmilestone-inin16042 architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042
2016-12-20 21:02:14 Kevin W. Rudd affects ubuntu systemd (Ubuntu)
2017-01-05 16:38:05 Luciano Chavez systemd (Ubuntu): assignee Taco Screen team (taco-screen-team)
2017-01-06 01:09:53 Steve Langasek nominated for series Ubuntu Xenial
2017-01-06 01:09:53 Steve Langasek bug task added systemd (Ubuntu Xenial)
2017-01-06 01:09:53 Steve Langasek nominated for series Ubuntu Yakkety
2017-01-06 01:09:53 Steve Langasek bug task added systemd (Ubuntu Yakkety)
2017-01-07 01:44:33 Steve Langasek systemd (Ubuntu Xenial): assignee Steve Langasek (vorlon)
2017-01-07 01:44:35 Steve Langasek systemd (Ubuntu Xenial): milestone ubuntu-16.04.2
2017-01-07 01:44:40 Steve Langasek systemd (Ubuntu Yakkety): assignee Steve Langasek (vorlon)
2017-01-07 01:44:48 Steve Langasek systemd (Ubuntu Yakkety): milestone yakkety-updates
2017-01-08 02:19:41 bugproxy attachment added infinity parsing patch https://bugs.launchpad.net/bugs/1651518/+attachment/4801480/+files/systemd_229.patch
2017-01-08 02:19:43 bugproxy attachment added debdiff between systemd_229-4ubuntu14 and test build systemd_229-4ubuntu15 https://bugs.launchpad.net/bugs/1651518/+attachment/4801481/+files/systemd_infinity.debdiff
2017-01-23 17:38:24 Manoj Iyer systemd (Ubuntu): assignee Taco Screen team (taco-screen-team)
2017-03-21 05:13:54 Steve Langasek description == Comment: #1 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2016-12-19 04:15:10 == Configuration: IBM 8001-22C (S822LC), LSI SAS adapters, SMC 4U90 disk drawers, HDD (180) 7.3TB Problem: HTX exercisers stopped on error, with HTX log showing "rc 11, errno 11 from main(): pthread_create" htxubuntu-425 lpar: busybee.aus.stglabs.ibm.com (root/ lab passwd) root@busybee:~# uname -a Linux busybee 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:27:59 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux root@busybee:~# cat /tmp/htxerr /dev/sdh Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdh Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sdao Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdao Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sddx Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sddx Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sdcz Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdcz Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sddp Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sddp Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error No errors logged in syslog after starting HTX: ==== State: Open by: asperez on 16 December 2016 10:28:02 ==== This error recreated on the smaller 1U Open Power system with the same smaller 1-adapter/1-4U90 drawer/90 HDD. There are 2 cables connected to the drawer (one to each ESM) that requires multipath enabled. lpar: yellowbee root@yellowbee:~# cat /tmp/htxerr /dev/sdao Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdak Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdt Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdaz Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdn Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdv Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdaj Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdal Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create ==== State: Open by: cde00 on 19 December 2016 02:48:46 ==== This defect will go to Linux as even after making the below 2 changes in systemd resource limit, errors are seen: root@yellowbee:/etc/systemd/logind.conf.d# cat htxlogindcustom.conf [Login] UserTasksMax=infinity root@yellowbee:/etc/systemd/logind.conf.d# cat ../system.conf.d/htxsystemdcustom.conf [Manager] DefaultTasksAccounting=yes DefaultTasksMax=infinity root@yellowbee:/etc/systemd/logind.conf.d# logind limit for MaxUserTask as well as systemd limit was made infinite. Please look defect SW363655 for more details and the suggestion given earlier by Linux team. errors are still seen. So, would ask Linux team to take a look and see if anything else is causing these errors. == Comment: #3 - Kevin W. Rudd <ruddk@us.ibm.com> - 2016-12-19 17:34:27 == It appears that the version of logind on this system does not support the value of "infinity", and is reverting to the default of 12288: # cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max 12288 As a workaround until this can be resolved, specify an exact value. You can try using the current system thread-max value: # cat /proc/sys/kernel/threads-max 3974272 /etc/systemd/logind.conf.d/htxlogindcustom.conf: [Login] UserTasksMax=3974272 == Comment: #4 - Kevin W. Rudd - 2016-12-20 10:25:09 == Canonical, This issue appears to map to the following systemd bug and patch: https://github.com/systemd/systemd/issues/3833 https://github.com/systemd/systemd/commit/f50582649f8eee73f59aff95fadd9a963ed4ffea This patch appears to be included in debian/232-7, but is missing in the xenial and yakkety versions. [SRU justification] Before systemd 232, UserTasksMax=infinity is not respected in logind.conf, despite the documentation referring to systemd.resource-control(5) for the definition of this field. This limits the use of Ubuntu 16.04 and later in contexts where a user session should be permitted to allocate large numbers of processes. [Test case] 1. Set UserTasksMax=infinity in /etc/systemd/logind.conf. 2. Create a new login session. 3. Check the ulimit for user processes with 'ulimit -u'. 4. Verify that the limit has a numeric value. 5. Upgrade to systemd from -proposed. 6. Create a new login session. 7. Check the ulimit for user processes with 'ulimit -u'. 8. Verify that the limit is now set to 'unlimited'. [Regression potential] This is an upstream patch which is part of 232 and later without issues and clearly addresses the bug in question. While the upstream commit includes code changes that are not strictly required in order to fix this bug, these are mostly cosmetic and should not carry significant additional risk. == Comment: #1 - Application Cdeadmin <cdeadmin@us.ibm.com> - 2016-12-19 04:15:10 == Configuration: IBM 8001-22C (S822LC), LSI SAS adapters, SMC 4U90 disk drawers, HDD (180) 7.3TB Problem: HTX exercisers stopped on error, with HTX log showing "rc 11, errno 11 from main(): pthread_create" htxubuntu-425 lpar: busybee.aus.stglabs.ibm.com (root/ lab passwd) root@busybee:~# uname -a Linux busybee 4.4.0-51-generic #72-Ubuntu SMP Thu Nov 24 18:27:59 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux root@busybee:~# cat /tmp/htxerr /dev/sdh Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdh Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sdao Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdao Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sddx Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sddx Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sdcz Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdcz Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error /dev/sddp Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sddp Dec 12 23:52:42 2016 err=0000000b sev=1 hxestorage Hardware Exerciser stopped on an error No errors logged in syslog after starting HTX: ==== State: Open by: asperez on 16 December 2016 10:28:02 ==== This error recreated on the smaller 1U Open Power system with the same smaller 1-adapter/1-4U90 drawer/90 HDD. There are 2 cables connected to the drawer (one to each ESM) that requires multipath enabled. lpar: yellowbee root@yellowbee:~# cat /tmp/htxerr /dev/sdao Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdak Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdt Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdaz Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdn Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdv Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdaj Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create /dev/sdal Dec 16 01:14:44 2016 err=0000000b sev=1 hxestorage rc 11, errno 11 from main(): pthread_create ==== State: Open by: cde00 on 19 December 2016 02:48:46 ==== This defect will go to Linux as even after making the below 2 changes in systemd resource limit, errors are seen: root@yellowbee:/etc/systemd/logind.conf.d# cat htxlogindcustom.conf [Login] UserTasksMax=infinity root@yellowbee:/etc/systemd/logind.conf.d# cat ../system.conf.d/htxsystemdcustom.conf [Manager] DefaultTasksAccounting=yes DefaultTasksMax=infinity root@yellowbee:/etc/systemd/logind.conf.d# logind limit for MaxUserTask as well as systemd limit was made infinite. Please look defect SW363655 for more details and the suggestion given earlier by Linux team. errors are still seen. So, would ask Linux team to take a look and see if anything else is causing these errors. == Comment: #3 - Kevin W. Rudd <ruddk@us.ibm.com> - 2016-12-19 17:34:27 == It appears that the version of logind on this system does not support the value of "infinity", and is reverting to the default of 12288: # cat /sys/fs/cgroup/pids/user.slice/user-0.slice/pids.max 12288 As a workaround until this can be resolved, specify an exact value. You can try using the current system thread-max value: # cat /proc/sys/kernel/threads-max 3974272 /etc/systemd/logind.conf.d/htxlogindcustom.conf: [Login] UserTasksMax=3974272 == Comment: #4 - Kevin W. Rudd - 2016-12-20 10:25:09 == Canonical, This issue appears to map to the following systemd bug and patch: https://github.com/systemd/systemd/issues/3833 https://github.com/systemd/systemd/commit/f50582649f8eee73f59aff95fadd9a963ed4ffea This patch appears to be included in debian/232-7, but is missing in the xenial and yakkety versions.
2017-03-24 16:09:53 Łukasz Zemczak systemd (Ubuntu): status New Fix Released
2017-04-11 08:19:20 Łukasz Zemczak systemd (Ubuntu Yakkety): status New Fix Committed
2017-04-11 08:19:22 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2017-04-11 08:19:23 Łukasz Zemczak bug added subscriber SRU Verification
2017-04-11 08:19:27 Łukasz Zemczak tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-needed
2017-04-13 21:30:52 bugproxy tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-needed architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety
2017-04-21 04:00:28 Launchpad Janitor systemd (Ubuntu Yakkety): status Fix Committed Fix Released
2017-04-21 04:00:45 Steve Langasek removed subscriber Ubuntu Stable Release Updates Team
2017-05-08 19:38:35 Manoj Iyer bug task added ubuntu-power-systems
2017-05-09 14:47:59 Manoj Iyer ubuntu-power-systems: status New Fix Committed
2017-06-07 18:10:52 Steve Langasek systemd (Ubuntu Xenial): assignee Steve Langasek (vorlon) Dimitri John Ledkov (xnox)
2017-06-07 21:32:35 Dimitri John Ledkov systemd (Ubuntu Xenial): importance Undecided High
2017-06-07 21:32:41 Dimitri John Ledkov systemd (Ubuntu Xenial): milestone ubuntu-16.04.2 ubuntu-16.04.3
2017-07-10 15:32:31 Łukasz Zemczak systemd (Ubuntu Xenial): status New Fix Committed
2017-07-10 15:32:33 Łukasz Zemczak bug added subscriber Ubuntu Stable Release Updates Team
2017-07-10 15:32:39 Łukasz Zemczak tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-needed verification-needed-xenial
2017-07-10 15:49:51 bugproxy bug watch added https://github.com/systemd/systemd/issues/3833
2017-07-11 13:21:09 Dimitri John Ledkov tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-needed verification-needed-xenial architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-failed-xenial verification-needed
2017-07-12 11:45:13 Dimitri John Ledkov tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-failed-xenial verification-needed architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done verification-done-xenial verification-done-yakkety
2017-07-18 23:33:44 Adam Conrad tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done verification-done-xenial verification-done-yakkety architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-needed verification-needed-xenial
2017-07-19 13:38:28 Dimitri John Ledkov tags architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done-yakkety verification-needed verification-needed-xenial architecture-ppc64le bot-comment bugnameltc-150030 severity-high targetmilestone-inin16042 verification-done verification-done-xenial verification-done-yakkety
2017-07-19 16:47:21 Manoj Iyer ubuntu-power-systems: importance Undecided High
2017-07-20 23:43:25 Launchpad Janitor systemd (Ubuntu Xenial): status Fix Committed Fix Released
2017-07-20 23:43:25 Launchpad Janitor cve linked 2017-9445
2017-09-04 13:37:37 Andrew Cloke ubuntu-power-systems: status Fix Committed Fix Released