lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: AppArmor denies /dev/ptmx mounting
| Affects | Status | Importance | Assigned to | Milestone | ||
|---|---|---|---|---|---|---|
| lxc (Ubuntu) | ||||||
| | Precise |
High
|
Unassigned | |||
| | Trusty |
High
|
Unassigned | |||
Bug Description
=======
SRU Justification (for precise):
Impact: containers fail to start!
Regression potential: we only add a copy of an existing apparmor allow rule
with a different syntax (no trailing /), leaving the old one for older
aa/kernel. There should be no regressions.
Test case: lxc-create -t ubuntu -n u1; lxc-start -n u1
=======
We are seeing test suite failures under ADT testing with linux, linux-lts-utopic and linux-lts-vivid kernels:
https:/
https:/
https:/
https:/
https:/
https:/
| Stéphane Graber (stgraber) wrote : | #1 |
| Martin Pitt (pitti) wrote : | #2 |
> or a failure to reach the cloud image server.
The tests work on wily and vivid, so in principle they can talk to the cloud image server or linuxcontainers
The main difference in http://
Anyway, I'll investigate this more closely and follow up here.
| Martin Pitt (pitti) wrote : | #3 |
Keeping notes: I did a local QEMU run against trusty release and trusty-proposed:
adt-run lxc -s --- qemu /srv/vm/
adt-run --apt-pocket=
They both fail for the same reason: five tests fail due to "ERROR: Unable to fetch GPG key from keyserver." -- presumably because the test has some special magic with "Running in the Canonical CI environment" which doesn't apply to my laptop where no proxy is in use. The test doesn't hang there, but that doesn't say that much as the test apparently behaves rather different in local qemu vs. Canonical cloud with proxy.
| Martin Pitt (pitti) wrote : | #4 |
Running lxc test against trusty-release in the CI production environment still works fine (against kernel -65). I do get the hang with running against -proposed, under otherwise the exact same circumstances.
The dist-upgrade to -proposed does the following:
The following NEW packages will be installed:
linux-
linux-
The following packages will be upgraded:
apport grub-common grub-pc grub-pc-bin grub2-common libpam-systemd
libpython3.
libsystemd-login0 libudev1 linux-headers-
linux-
python3-
I obviously hangs in lxc-test-ubuntu. I wonder if that's the first test which actually uses a bootstrapped full ubuntu image, not just a simple busybox one?
When it hangs, the following test related processes are running:
lxc-dns+ 3298 0.0 0.0 28204 956 ? S 09:07 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=
root 12758 0.0 0.0 4440 656 ? S 09:11 0:00 /bin/sh /usr/bin/
root 31374 0.0 0.0 34724 1348 ? Ss 09:13 0:00 /usr/lib/
root 31426 0.0 0.0 34712 1504 ? S 09:13 0:00 lxc-wait -n 4a5f2adb-
$ sudo lxc-ls -f
NAME STATE IPV4 IPV6 AUTOSTART
-------
4a5f2adb-
so it seems the container never starts up? /var/lib/
However, dmesg contains
[ 352.395653] type=1400 audit(144455481
and when I try to start it, I indeed get
root@adt:~# lxc-start -n 4a5f2adb-
lxc-start: conf.c: setup_pts: 1772 Permission denied - mount failed '/dev/pts/
lxc-start: conf.c: lxc_setup: 4230 failed to setup the new pts instance
lxc-start: start.c: do_start: 688 failed to setup the container
lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2
lxc-start: start.c: __lxc_start: 1080 failed to spawn '4a5f2adb-
lxc-start: lxc_start.c: main: 342 The container failed to start.
lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options.
and the same apparmor error repeated. So this surely does look like some lxc/kern...
| summary: |
- lxc: test suites failing on trusty + lxc-test-ubuntu hangs forever in trusty-proposed |
To completely rule out that it's not the python3.4 regression in trusty-proposed (bug 1500768) or the (really unrelated) udev fix in bug 1470399 I instead ran it with --apt-pocket=
@Stéphane: It would be really useful if the test suite could detect failures of lxc-start, and show its output on failures. Such quiet/forever-
| summary: |
- lxc-test-ubuntu hangs forever in trusty-proposed + lxc-test-ubuntu hangs forever in trusty-proposed: AppArmor denies + /dev/ptmx mounting |
| summary: |
- lxc-test-ubuntu hangs forever in trusty-proposed: AppArmor denies - /dev/ptmx mounting + lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: + AppArmor denies /dev/ptmx mounting |
| Changed in linux (Ubuntu): | |
| status: | New → Invalid |
| Changed in lxc (Ubuntu): | |
| status: | New → Invalid |
| tags: | added: regression-proposed |
| Stéphane Graber (stgraber) wrote : | #6 |
What I don't get is why the other tests aren't failing too, they all start containers too and so should hit the exact same failure. Why one of the last tests is the one hanging just doesn't make sense to me.
Anyway, looks like there's a way for us to reproduce this and look into it. It may well be a kernel/apparmor regression after all.
| Martin Pitt (pitti) wrote : | #7 |
I suppose the recent kernel patch
UBUNTU: SAUCE: (no-up) apparmor: fix mount not handling disconnected paths
which got backported to trusty causes this regression. As the same code is present in later releases, I guess that in v/w lxc has an updated apparmor profile which allows the operation that trusty's lxc is now failing on due to the new apparmor violation?
| John Johansen (jjohansen) wrote : | #8 |
yes,
UBUNTU: SAUCE: (no-up) apparmor: fix mount not handling disconnected paths
is causing the regression. However reverting this fix will cause issues for Bug 1496430, which was blocking a fix for a CVE.
The correct solution is to update the profile.
Hello Andy, or anyone else affected,
Accepted lxc into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
| Changed in lxc (Ubuntu Trusty): | |
| status: | New → Fix Committed |
| tags: | added: verification-needed |
| John Johansen (jjohansen) wrote : | #10 |
To be specific I added the rule
mount options=(rw,bind) /dev/pts/ptmx -> /dev/ptmx,
to the lxc-start profile
| Martin Pitt (pitti) wrote : | #11 |
Hello Andy, or anyone else affected,
Accepted lxc into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
| Launchpad Janitor (janitor) wrote : | #12 |
Status changed to 'Confirmed' because the bug affects multiple users.
| Changed in linux (Ubuntu Trusty): | |
| status: | New → Confirmed |
| Stephen Gaito (3-stephen) wrote : | #13 |
Hello,
As of this morning's security roll out of the Linux 3.13.0-66 kernel, this bug *is* effecting *live* LXC containers ;-(
(I am using Trusty 14.04 LTS - I note that recently built Trusty 14.04.3 machines are not rolling out Linux 3.13.0-66 as they have Linux 3.19.0-30-generic)
Reading between the lines of @jjohansen's comments (#8, #10), I updated my /etc/apparmor.
> mount options=bind /dev/pts/ptmx/ -> /dev/ptmx/,
to
> mount options=(rw, bind) /dev/pts/ptmx/ -> /dev/ptmx/,
Unfortunately I can confirm that this *does* *not* solve the problem. Have I misunderstood something?
I can also confirm that I have exactly the same error messages listed above in @pitti's comment #4.
Is there any know work-around/fix?
| Daniel Bull (ubuntu-frozenmist) wrote : | #14 |
I can confirm the same as what Stephen just said.
Servers rebooted overnight for the security patch and none of the LXC containers restarted or can be started.
lxc-start: conf.c: setup_pts: 1772 Permission denied - mount failed '/dev/pts/
This is on a live 14.04 LTS server
Again I can confirm modifying the start-container file does not work
| Nicolas Delvaux (malizor) wrote : | #15 |
@Stephen: I had the same problem after today's upgrade.
Activating the proposed repository and upgrading lxc to version 1.0.7-0ubuntu0.9 fixed the issue for me.
See comment #11 for details.
But it's a shame this proposed fix was not released to everyone before the new kernel.
| tags: |
added: verification-done removed: verification-needed |
| Daniel Bull (ubuntu-frozenmist) wrote : | #16 |
This seems to explain it, currently trying to teach myself apparmor to find a temporary fix...
apps kernel: [ 707.036112] audit: type=1400 audit(144533185
| Mark Thornton (mthornton-2) wrote : | #17 |
The proposed update works for us. When is it likely to be released as we don't want to do this on our production servers?
| Martin Pitt (pitti) wrote : | #18 |
I'm expediting the usual 7 day maturing period; this is a rather grave regression and apparently the new kernel didn't get around to add a Breaks: to the previous LXC version. Thanks for verifying!
| Launchpad Janitor (janitor) wrote : | #19 |
This bug was fixed in the package lxc - 1.0.7-0ubuntu0.9
---------------
lxc (1.0.7-0ubuntu0.9) trusty; urgency=medium
* Update previous patch to include some extra apparmor rules.
(LP: #1504781)
-- Stéphane Graber <email address hidden> Wed, 14 Oct 2015 13:59:48 -0700
| Changed in lxc (Ubuntu Trusty): | |
| status: | Fix Committed → Fix Released |
| Martin Pitt (pitti) wrote : Update Released | #20 |
The verification of the Stable Release Update for lxc has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
| Changed in linux (Ubuntu Trusty): | |
| status: | Confirmed → Invalid |
| Alex L. Demidov (alexeydemidov) wrote : | #21 |
Is there a fix for Ubuntu 12.04 LTS ?
| Stephen Gaito (3-stephen) wrote : | #22 |
@Martin, many thanks for releasing this into the "wild".
I can confirm that it has now appeared on Trusty-updates on 1&1 servers and in the "normal" GB archives.
I can also confirm that this fixes my LXC server problems.
| Daniel (hackie) wrote : | #23 |
I see a connection to https:/
| Jan Groenewald (jan-aims) wrote : Re: [Aims] [Bug 1504781] Re: lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: AppArmor denies /dev/ptmx mounting | #24 |
Looks like it is a duplicate. I've marked it as such.
Regards
Jan
On 20 October 2015 at 20:28, Daniel <email address hidden> wrote:
> I see a connection to
> https:/
> duplicate?
>
> --
> You received this bug notification because you are a member of AIMS,
> which is subscribed to the bug report.
> https:/
>
> Title:
> lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66:
> AppArmor denies /dev/ptmx mounting
>
> Status in linux package in Ubuntu:
> Invalid
> Status in lxc package in Ubuntu:
> Invalid
> Status in linux source package in Trusty:
> Invalid
> Status in lxc source package in Trusty:
> Fix Released
>
> Bug description:
> We are seeing test suite failures under ADT testing with linux, linux-
> lts-utopic and linux-lts-vivid kernels:
>
>
> https:/
>
> https:/
>
>
> https:/
>
> https:/
>
>
> https:/
>
> https:/
>
> To manage notifications about this bug go to:
> https:/
>
> --
> Mailing list: https:/
> Post to : <email address hidden>
> Unsubscribe : https:/
> More help : https:/
>
--
.~.
/V\ Jan Groenewald
/( )\ www.aims.ac.za
^^-^^
| Paul Sokolovsky (pfalcon) wrote : | #25 |
I got hit by the same issue, with the same unlucky kernel (installed from normal update channel):
root@x230:~# uname -a
Linux x230 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Will read suggestions above before going to back to classical VM...
| Paul Sokolovsky (pfalcon) wrote : | #26 |
The above report is against:
lxc 1.0.7-0ubuntu0.7 amd64
| Jan Groenewald (jan-aims) wrote : | #27 |
The fix is already released. Update lxc, then update the kernel.
Regards,
Jan
On 22 October 2015 at 17:44, Paul Sokolovsky <email address hidden>
wrote:
> I got hit by the same issue, with the same unlucky kernel (installed
> from normal update channel):
>
> root@x230:~# uname -a
> Linux x230 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
> Will read suggestions above before going to back to classical VM...
>
> --
> You received this bug notification because you are a member of AIMS,
> which is subscribed to the bug report.
> https:/
>
> Title:
> lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66:
> AppArmor denies /dev/ptmx mounting
>
> Status in linux package in Ubuntu:
> Invalid
> Status in lxc package in Ubuntu:
> Invalid
> Status in linux source package in Trusty:
> Invalid
> Status in lxc source package in Trusty:
> Fix Released
>
> Bug description:
> We are seeing test suite failures under ADT testing with linux, linux-
> lts-utopic and linux-lts-vivid kernels:
>
>
> https:/
>
> https:/
>
>
> https:/
>
> https:/
>
>
> https:/
>
> https:/
>
> To manage notifications about this bug go to:
> https:/
>
> --
> Mailing list: https:/
> Post to : <email address hidden>
> Unsubscribe : https:/
> More help : https:/
>
--
.~.
/V\ Jan Groenewald
/( )\ www.aims.ac.za
^^-^^
| Paul Sokolovsky (pfalcon) wrote : | #28 |
Upgrading to 1.0.7-0ubuntu0.9 from updates fixed it. Sorry for the noise.
| Stratos Zolotas (baskin) wrote : | #29 |
Another one has asked but no reply yet. Is a fix for 12.04 going to be released? The bug is still valid there.
| Serge Hallyn (serge-hallyn) wrote : Re: [Bug 1504781] Re: lxc-test-ubuntu hangs forever in trusty-proposed with Linux 3.13.0-66: AppArmor denies /dev/ptmx mounting | #30 |
Quoting Stratos Zolotas (<email address hidden>):
> Another one has asked but no reply yet. Is a fix for 12.04 going to be
> released? The bug is still valid there.
Which bug are you looking for? You're using a backport or ppa
or custom built lxc and are looking for a kernel fix?
| Eugene Miloslavsky (eugenemi) wrote : | #31 |
as documented in comment 20 or https:/
| Stratos Zolotas (baskin) wrote : | #32 |
As Eugene said, there is a bug affecting precise with the latest 3.2 and 3.13 (supported LTS trusty kernel for precise) and the official LXC package. No backports or ppa used.
The bug is marked as duplicate but no fix for 12.04 is released.
| Serge Hallyn (serge-hallyn) wrote : | #33 |
Judging by jjohansen's comment #8, I guess the shipped common configuration files in precise's lxc should be updated to include the new rule. Precise's lxc is in universe, community supported. Can you provide a proposed, tested debdiff and ping me? I'll sponsor it when ready if needed.
| DawnCold (loooseleaves) wrote : | #34 |
+1
after upgraded to lxc 1.0.7-0ubuntu0.9 it works for me
| Robert (baumgaro) wrote : | #35 |
For me in 12.04 this worked:
Adding the PPA - https:/
| Marc Reymann (mreymann) wrote : | #36 |
So, there's still no fix for 12.04 in the standard repos? I mean, Precise is called "LTS" after all.
| Stefan Huehner (stefan-huehner) wrote : | #37 |
Hi Marc,
please check comment #33 from Serge. He explained that formally lxc in precise is not covered by lts.
However a very special case as a LTS update in main in precise (kernel) did break unrelated software (lxc) which is a clear regression.
And then not having that other software not in being in main -> skips it from LTS is lets say very annoying.
@Robert:
using that higher version of lxc is not a perfect drop-up replacement either as usage changed compared to old precise lxc (i.e. no lxc-list anymore but lxc-ls -f) and maybe other changes.
We'll probably look into backporting the lxc fix for precise as we have quite a few machines affected. But no ETA, so if anybody else here can pick that up before that would be very welcome.
@Serge:
Any chance to get some policy statement from Ubuntu here? As i see maybe that example as very special regression caused by lts update should maybe warrant fixing items not in main if broken by it.
| Serge Hallyn (serge-hallyn) wrote : | #38 |
@stefan-huehner - sorry, I'm losing track. is what you are asking for just a lxc update to precise-proposed with the new apparmor allow rule that jj suggested?
If so, in comment #33 I was trying to encourage a debdiff to be posted by someone who could best test it. I'll then sponsor it into the archive.
I'll make a note in my tickler file that if noone has posted a debdiff by friday, I'll post one then.
| Mathieu Lafon (mlafon) wrote : | #39 |
I'm also interested by an update of the lxc package for precise.
The attached patch is working for me (add "/dev/pts/ptmx -> /dev/ptmx" instead of "/dev/pts/ptmx/ -> /dev/ptmx/"). Note that keeping the previous rule is required for not breaking old kernels.
| description: | updated |
| Changed in lxc (Ubuntu Precise): | |
| importance: | Undecided → High |
| Changed in linux (Ubuntu Precise): | |
| importance: | Undecided → High |
| description: | updated |
| Stefan Huehner (stefan-huehner) wrote : | #40 |
We have tested the patch from #39 by applying in manually in on of our affected systems and can confirm that it fixes the regression. With it in place lxc-start works again when having latest precise 3.2 kernel.
@Mathieu:
Thanks for providing it.
| Serge Hallyn (serge-hallyn) wrote : | #41 |
Hi,
The fix was uploaded last week for acceptance by the SRU team. It's
waiting to be accepted into -proposed. Then it will need to be tested
to be accepted into -updates.
https:/
Hello Andy, or anyone else affected,
Accepted lxc into precise-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-
Further information regarding the verification process can be found at https:/
| Changed in lxc (Ubuntu Precise): | |
| status: | New → Fix Committed |
| tags: | removed: verification-done |
| tags: | added: verification-needed |
| no longer affects: | linux (Ubuntu Trusty) |
| no longer affects: | linux (Ubuntu Precise) |
| no longer affects: | linux (Ubuntu) |
| no longer affects: | lxc (Ubuntu) |
| Changed in lxc (Ubuntu Trusty): | |
| importance: | Undecided → High |
| Stefan Huehner (stefan-huehner) wrote : | #43 |
Hello Stephane,
we have tested the propose 0.7.5-3ubuntu70 package in precise.
We have verified that with latest 3.2.0 kernel having the regression (linux-
We have also verified that installing same 0.7.5-3ubuntu70 from proposed with older kernel linux-headers-
Note:
In all cases we noticed another apparmor DENIED entry on lxc-shutdown (both old+new kernel) however it seems to not directly affect functionality and not be directly related to this bug (just for info here):
[2459430.608467] type=1400 audit(144674485
| tags: |
added: verification-done removed: verification-needed |
| Christoph Mitasch (cmitasch) wrote : | #44 |
Hello,
I can also confirm that lxc_0.7.
Christoph
| Launchpad Janitor (janitor) wrote : | #45 |
This bug was fixed in the package lxc - 0.7.5-3ubuntu70
---------------
lxc (0.7.5-3ubuntu70) precise-proposed; urgency=medium
* d/lxc.apparmor: add ptmx bind mount rule with different syntax to work
around a regression in the aa parser. (LP: #1504781)
-- Serge Hallyn <email address hidden> Wed, 28 Oct 2015 09:06:26 -0500
| Changed in lxc (Ubuntu Precise): | |
| status: | Fix Committed → Fix Released |
They all get stuck in lxc-test-ubuntu which would indicate either a hang in debootstrap (newly introduced debconf question) or a failure to reach the cloud image server.
In either case, you've not actually regressed LXC, the other tests would have failed if that was the case.
So I'd toss this one over to pitti for investigation and release the updated kernels regardless.