lxc-net fails to start with bash

[Robert Blenis]

=======================================
SRU Justification:
1. Impact: lxc-net fails to start if /bin/sh is linked to bash
2. Development fix: Fix the call to 'cleanup()' which does not work in bash
3. Stable fix: same as development fix
4. Test case:
   1. install lxc
   2. Link /bin/sh to /bin/bash
   3. 'sudo stop lxc; sudo start lxc'
   This will be added to the lxc testsuite
5. Regresion potential: a '()' is (apparently) allowed in dash, but not
   required. This is purely a typo fix and should have no potential for
   regression.
=======================================

ran:
  sudo apt-get install mdadm
and got the following:
---------------------------------------------------------
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libglew1.5 libglewmx1.5
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  mdadm
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 360 kB of archives.
After this operation, 863 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main mdadm amd64 3.2.3-2ubuntu1 [360 kB]
Fetched 360 kB in 1s (247 kB/s)
Preconfiguring packages ...
dpkg: warning: parsing file '/var/lib/dpkg/available' near line 47141 package 'handbrake-gtk':
 error in Version string 'svn3918ppa1~maverick1': version number does not start with digit
Selecting previously unselected package mdadm.
(Reading database ... 270405 files and directories currently installed.)
Unpacking mdadm (from .../mdadm_3.2.3-2ubuntu1_amd64.deb) ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Processing triggers for doc-base ...
Processing 6 added doc-base files...
Registering documents with scrollkeeper...
Processing triggers for man-db ...
Setting up lxc (0.7.5-3ubuntu54) ...
start: Job failed to start
invoke-rc.d: initscript lxc-net, action "start" failed.
dpkg: error processing lxc (--configure):
 subprocess installed post-installation script returned error exit status 1
Setting up mdadm (3.2.3-2ubuntu1) ...
No apport report written because MaxReports is reached already
                                                              Generating mdadm.conf... done.
 Removing any system startup links for /etc/init.d/mdadm-raid ...
update-initramfs: deferring update (trigger activated)
 * Starting MD monitoring service mdadm --monitor [ OK ]
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.2.0-24-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Errors were encountered while processing:
 lxc
E: Sub-process /usr/bin/dpkg returned an error code (1)
------------------------------------------------------------------------------

after which I cleared out the /var/crash directory and reran: sudo apt-get install mdadm, and got:
----------------------------------------------------------------------------
Reading package lists... Done
Building dependency tree
Reading state information... Done
mdadm is already the newest version.
The following packages were automatically installed and are no longer required:
  libglew1.5 libglewmx1.5
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue [Y/n]? y
Setting up lxc (0.7.5-3ubuntu54) ...
start: Job failed to start
invoke-rc.d: initscript lxc-net, action "start" failed.
dpkg: error processing lxc (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 lxc
E: Sub-process /usr/bin/dpkg returned an error code (1)

ProblemType: Package
DistroRelease: Ubuntu 12.04
Package: lxc 0.7.5-3ubuntu54
ProcVersionSignature: Ubuntu 3.2.0-24.37-generic 3.2.14
Uname: Linux 3.2.0-24-generic x86_64
ApportVersion: 2.0.1-0ubuntu7
AptOrdering: lxc: Configure
Architecture: amd64
Date: Wed May 16 06:49:27 2012
DpkgTerminalLog:
 Setting up lxc (0.7.5-3ubuntu54) ...
 start: Job failed to start
 invoke-rc.d: initscript lxc-net, action "start" failed.
 dpkg: error processing lxc (--configure):
  subprocess installed post-installation script returned error exit status 1
DuplicateSignature:
 Setting up lxc (0.7.5-3ubuntu54) ...
 start: Job failed to start
 invoke-rc.d: initscript lxc-net, action "start" failed.
 dpkg: error processing lxc (--configure):
  subprocess installed post-installation script returned error exit status 1
ErrorMessage: subprocess installed post-installation script returned error exit status 1
InstallationMedia: Ubuntu 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.1)
SourcePackage: lxc
Title: package lxc 0.7.5-3ubuntu54 failed to install/upgrade: subprocess installed post-installation script returned error exit status 1
UpgradeStatus: Upgraded to precise on 2012-04-27 (18 days ago)

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug. Interesting - so installing mdadm stopped lxc from starting? I can't reproduce that here, unfortunately.

Do you have autostart containers defined? (what does ls -l /etc/lxc/auto show?)

Can you add the line:

console output

right underneath the 'stop on' lines in both /etc/init/lxc.conf and /etc/init/lxc-net.conf, then do

stop lxc
start lxc

and show the output here?

[Serge Hallyn (serge-hallyn)]

(marking incomplete while awaiting response to get more information to debug)

[Serge Hallyn (serge-hallyn)]

(note - actually dmesg shows that lxc-net is the one failing to start)

[Robert Blenis (rblenis)]

 I don't know if lxc was previously working or not. Another person has used
this computer recently, and I don't know what they may have done to it. I
remember installing lxc some time ago, but never had time to actually do
anything with it.

On 05/16/2012 11:04 AM, Serge Hallyn wrote:

Thanks for reporting this bug. Interesting - so installing mdadm
stopped lxc from starting? I can't reproduce that here, unfortunately.

Do you have autostart containers defined? (what does ls -l
/etc/lxc/auto show?)

total 0

 Can you add the line:

console output

right underneath the 'stop on' lines in both /etc/init/lxc.conf and
/etc/init/lxc-net.conf, then do

stop lxc
start lxc

and show the output here?

rblenis@rblenis-laptop:/etc/init$ stop lxc stop: Rejected send message, 1
matched rules; type="method_call", sender=":1.73" (uid=1000 pid=2862
comm="stop lxc ") interface="com.ubuntu.Upstart0_6.Job" member="Stop" error
name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart" (uid=0
pid=1 comm="/sbin/init")

rblenis@rblenis-laptop:/etc/init$ start lxc start: Rejected send message, 1
matched rules; type="method_call", sender=":1.74" (uid=1000 pid=2870
comm="start lxc ") interface="com.ubuntu.Upstart0_6.Job" member="Start"
error name="(unset)" requested_reply="0" destination="com.ubuntu.Upstart"
(uid=0 pid=1 comm="/sbin/init")

** Changed in: lxc (Ubuntu)
   Importance: Undecided => Medium

** Changed in: lxc (Ubuntu)
       Status: New => Incomplete

[Robert Blenis (rblenis)]

also, before installing mdadm, I did let a bunch of updates install, and
I think it may have complained then as well, but I had to get some work
done, and everything else seemed ok. Then when I needed mdadm to test
an external raid box, this is when I had time to investigate further.
sorry I don't have any further info on when the problem may have first
occurred.

On 05/16/2012 11:04 AM, Serge Hallyn wrote:
> Thanks for reporting this bug. Interesting - so installing mdadm
> stopped lxc from starting? I can't reproduce that here, unfortunately.
>
> Do you have autostart containers defined? (what does ls -l
> /etc/lxc/auto show?)
>
> Can you add the line:
>
> console output
>
> right underneath the 'stop on' lines in both /etc/init/lxc.conf and
> /etc/init/lxc-net.conf, then do
>
> stop lxc
> start lxc
>
> and show the output here?
>
> ** Changed in: lxc (Ubuntu)
> Importance: Undecided => Medium
>
> ** Changed in: lxc (Ubuntu)
> Status: New => Incomplete
>

[Serge Hallyn (serge-hallyn)]

Sorry, these should have been

sudo lxc stop
sudo lxc start

If you still care to debug and provide that output, that would be great.
But if you aren't using lxc anyway, and simply wish to remove it, you
can do:

sudo apt-get purge lxc

You can also leave lxc installed, but stop it from starting its own
network by setting USE_LXC_BRIDGE=false in /etc/default/lxc.

If you remove lxc, please let us know so we can update the bug
status accordingly.

thanks,
-serge

[Robert Blenis (rblenis)]

On Wed, May 16, 2012 at 2:28 PM, Serge Hallyn <email address hidden>wrote:

> Sorry, these should have been
>
> sudo lxc stop
> sudo lxc start
>

rblenis@rblenis-laptop:/etc/init$ sudo stop lxc
lxc stop/waiting
rblenis@rblenis-laptop:/etc/init$ sudo start lxc
lxc start/running

>
> If you still care to debug and provide that output, that would be great.
> But if you aren't using lxc anyway, and simply wish to remove it, you
> can do:
>
> sudo apt-get purge lxc
>
> You can also leave lxc installed, but stop it from starting its own
> network by setting USE_LXC_BRIDGE=false in /etc/default/lxc.
>
> If you remove lxc, please let us know so we can update the bug
> status accordingly.
>
at some point when I have time, I do want to investigate using lxc instead
of/in addition to Virtualbox for our servers. currently this is just on a
spare development laptop, so if some other dev tool is conflicting, no big
deal, but I will have mdadm on the server, though I'm pretty sure the lxc
problem started before installing mdadm. I don't recall messing with any of
the config files when lxc was first installed, but my memory ain't what it
used to be. I have not tried uninstalling and reinstalling lxc to see if
that fixes anything. Let me know if there is anything else you want me to
try.

>
> thanks,
> -serge
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1000174
>
> Title:
> package lxc 0.7.5-3ubuntu54 failed to install/upgrade: subprocess
> installed post-installation script returned error exit status 1
>
> Status in “lxc” package in Ubuntu:
> Incomplete
>
> Bug description:
> ran:
> sudo apt-get install mdadm
> and got the following:
> ---------------------------------------------------------
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following packages were automatically installed and are no longer
> required:
> libglew1.5 libglewmx1.5
> Use 'apt-get autoremove' to remove them.
> The following NEW packages will be installed:
> mdadm
> 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
> 1 not fully installed or removed.
> Need to get 360 kB of archives.
> After this operation, 863 kB of additional disk space will be used.
> Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main mdadm amd64
> 3.2.3-2ubuntu1 [360 kB]
> Fetched 360 kB in 1s (247 kB/s)
> Preconfiguring packages ...
> dpkg: warning: parsing file '/var/lib/dpkg/available' near line 47141
> package 'handbrake-gtk':
> error in Version string 'svn3918ppa1~maverick1': version number does not
> start with digit
> Selecting previously unselected package mdadm.
> (Reading database ... 270405 files and directories currently installed.)
> Unpacking mdadm (from .../mdadm_3.2.3-2ubuntu1_amd64.deb) ...
> Processing triggers for ureadahead ...
> ureadahead will be reprofiled on next reboot
> Processing triggers for doc-base ...
> Processing 6 added doc-base files...
> Registering documents with scrollkeeper...
> Processing triggers for man-db ...
> Setting up lxc (0.7.5-3ubuntu54) ...
> start: Job failed to star...

[Serge Hallyn (serge-hallyn)]

Interesting so when you restarted lxc, it worked. what if you do

sudo stop lxc-net
sudo start lxc-net

?

[Robert Blenis (rblenis)]

should I remove the 'console output' line from the init config file first?

On 05/16/2012 09:15 PM, Serge Hallyn wrote:
> Interesting so when you restarted lxc, it worked. what if you do
>
> sudo stop lxc-net
> sudo start lxc-net
>
> ?
>

[Serge Hallyn (serge-hallyn)]

Quoting Robert Blenis (<email address hidden>):
> should I remove the 'console output' line from the init config file
> first?

No, it's what will hopefully show errors in your terminal.

[Robert Blenis (rblenis)]

rblenis@rblenis-laptop:/etc/init$ sudo stop lxc-net
stop: Unknown instance:
rblenis@rblenis-laptop:/etc/init$ sudo start lxc-net
start: Job failed to start

On Thu, May 17, 2012 at 8:41 AM, Serge Hallyn <email address hidden>wrote:

> Quoting Robert Blenis (<email address hidden>):
> > should I remove the 'console output' line from the init config file
> > first?
>
> No, it's what will hopefully show errors in your terminal.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1000174
>
> Title:
> package lxc 0.7.5-3ubuntu54 failed to install/upgrade: subprocess
> installed post-installation script returned error exit status 1
>
> Status in “lxc” package in Ubuntu:
> Incomplete
>
> Bug description:
> ran:
> sudo apt-get install mdadm
> and got the following:
> ---------------------------------------------------------
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following packages were automatically installed and are no longer
> required:
> libglew1.5 libglewmx1.5
> Use 'apt-get autoremove' to remove them.
> The following NEW packages will be installed:
> mdadm
> 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
> 1 not fully installed or removed.
> Need to get 360 kB of archives.
> After this operation, 863 kB of additional disk space will be used.
> Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main mdadm amd64
> 3.2.3-2ubuntu1 [360 kB]
> Fetched 360 kB in 1s (247 kB/s)
> Preconfiguring packages ...
> dpkg: warning: parsing file '/var/lib/dpkg/available' near line 47141
> package 'handbrake-gtk':
> error in Version string 'svn3918ppa1~maverick1': version number does not
> start with digit
> Selecting previously unselected package mdadm.
> (Reading database ... 270405 files and directories currently installed.)
> Unpacking mdadm (from .../mdadm_3.2.3-2ubuntu1_amd64.deb) ...
> Processing triggers for ureadahead ...
> ureadahead will be reprofiled on next reboot
> Processing triggers for doc-base ...
> Processing 6 added doc-base files...
> Registering documents with scrollkeeper...
> Processing triggers for man-db ...
> Setting up lxc (0.7.5-3ubuntu54) ...
> start: Job failed to start
> invoke-rc.d: initscript lxc-net, action "start" failed.
> dpkg: error processing lxc (--configure):
> subprocess installed post-installation script returned error exit status
> 1
> Setting up mdadm (3.2.3-2ubuntu1) ...
> No apport report written because MaxReports is reached already
> Generating
> mdadm.conf... done.
> Removing any system startup links for /etc/init.d/mdadm-raid ...
> update-initramfs: deferring update (trigger activated)
> * Starting MD monitoring service mdadm --monitor
> [ OK ]
> Processing triggers for initramfs-tools ...
> update-initramfs: Generating /boot/initrd.img-3.2.0-24-generic
> W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
> Errors were encountered while processing:
> lxc
> E: Sub-process /usr/bin/dpkg returned an error code (1)
>
> ---------------...

[Serge Hallyn (serge-hallyn)]

Quoting Robert Blenis (<email address hidden>):
> rblenis@rblenis-laptop:/etc/init$ sudo stop lxc-net
> stop: Unknown instance:
> rblenis@rblenis-laptop:/etc/init$ sudo start lxc-net
> start: Job failed to start

Since lxc started successfully, does that mean that you've set
USE_LXC_BRIDGE=false in /etc/default/lxc?

[Robert Blenis (rblenis)]

I did not change /etc/default/lxc.

On 05/17/2012 02:56 PM, Serge Hallyn wrote:
> Quoting Robert Blenis (<email address hidden>):
>> rblenis@rblenis-laptop:/etc/init$ sudo stop lxc-net
>> stop: Unknown instance:
>> rblenis@rblenis-laptop:/etc/init$ sudo start lxc-net
>> start: Job failed to start
> Since lxc started successfully, does that mean that you've set
> USE_LXC_BRIDGE=false in /etc/default/lxc?
>

[Serge Hallyn (serge-hallyn)]

Perplexing.

Please do so, to make sure you don't get future failures with 'start lxc' on boot or package upgrades. I'll mark this bug fix released for now, but if and when you start looking into using lxc and want to debug it, please feel free to reply and re-open this bug.

[Corey Schuhen (corey-d)]

Hi, I have the same issue. I played around and got a good way through the issue. Starting lxc itself is ok. It's starting lxc-net that is problematic. It is the starting of dnsmasq that is the issue.

It appears to be (at least in my case) caused by having /bin/sh set to bash.

If link /bin/sh to /bin/dash, all works fine. If I link it to /bin/bash, then I have the issue. Serge, can you see if you can reproduce this by setting your /bin/sh to /bin/bash?

[Serge Hallyn (serge-hallyn)]

Confirmed, that breaks it for me. Will fix that.

[Serge Hallyn (serge-hallyn)]

Note - I suspect the original poster's problem is a different one, but this one is worth fixing

[Serge Hallyn (serge-hallyn)]

I see - the problem is the '()' after 'cleanup()' toward the end of the pre-start

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 0.8.0~rc1-4ubuntu9

---------------
lxc (0.8.0~rc1-4ubuntu9) quantal; urgency=low

  * debian/lxc-net.upstart: don't put '()' after call to cleanup.
    (LP: #1000174)
 -- Serge Hallyn <email address hidden> Mon, 21 May 2012 08:26:25 -0700

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Clint Byrum (clint-fewbar)]

Hello Robert, or anyone else affected,

Accepted lxc into precise-proposed. The package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Stéphane Graber (stgraber)]

fix confirmed

[Launchpad Janitor (janitor)]

This bug was fixed in the package lxc - 0.7.5-3ubuntu58

---------------
lxc (0.7.5-3ubuntu58) precise-proposed; urgency=low

  * Fix broken logic in lxc-ubuntu template where lxc.devttydir would be
    set to 'lxc' only for releases that don't support it. (LP: #1007493)

lxc (0.7.5-3ubuntu57) precise-proposed; urgency=low

  [ Serge Hallyn ]
  * 0083-always-close-all-fds.patch: Have lxc-start always run with
    --close-all-fds. There is no advantage to having lxc-start fail with
    inherited fds. (LP: #1003583)
  * debian/lxc-net.upstart: don't put '()' after call to cleanup.
    (LP: #1000174)

  [ Stéphane Graber ]
  * Sync lxc-ubuntu with the one in Quantal:
    - Bugfixes:
      + Update list of extra packages for debootstrap to only include vim
        and ssh. The others were only relevant when we were still using the
        minbase variant. (LP: #996839)
      + Update default /etc/hosts to match that of a regular Ubuntu system.
        (adds missing ipv6 aliases) (LP: #1004108)
      + Make sure /etc/resolv.conf is valid before running any apt command.
        Fixes a potential race condition (no report of it at this time).

    - Improvements we get by pulling the whole patch from Quantal.
      These don't contain any user behaviour change but will make
      cherry-picking any further change much easier.
      + Drop any hardcoded Ubuntu version check and replace by feature
        checks instead. This removes the need for SRUs whenever we release
        a new Ubuntu.
      + Format lxc-ubuntu to consistently use 4-spaces indent instead
        of mixed spaces/tabs.
      + Update default /etc/network/interfaces to include the header.
      + Drop support for never supported releases (gutsy on sparc).
      + Update template help message for release and arch parameters.
        Old string was only listing i386 and amd64, which is no longer
        accurate (as of 12.04).
        (This string isn't translated)
      + Switch default Ubuntu version from lucid to precise for systems
        that don't have lsb_release (won't affect Ubuntu)

  * Sync lxc-start-ephemeral with the one in Quantal:
    - Switch lxc-start-ephemeral from unreliable parsing of DHCP lease files
      to using "ip netns" to retrieve the IP from the container's network
      namespace. (LP: #994752)
    - Fix a race in lxc-start-ephemeral where the container isn't yet
      running when trying to get its IPs.
    - Update a few calls so that lxc-start-ephemeral can be called as a
      user (ensure consistent usage of sudo across the script). (LP: #1004069)
 -- Stephane Graber <email address hidden> Fri, 01 Jun 2012 11:46:50 -0400