| 2013-08-19 18:34:16 |
Jay Taylor |
bug |
|
|
added bug |
| 2013-08-19 18:39:08 |
Jay Taylor |
description |
Using lxc with a btrfs filesystem can lead to system corruption and containers which are unable to boot when many containers are created and started simulatenously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu
`lsb_release -rd`:
ubuntu@ip-10-91-30-131:~$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg05472.html
And here is a key except regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is actually doing. It slowly launches 10 containers and then uses "&" to rapidly fork an additional 10 clone/start operations. I have it doing 2 cycles of this, and running the script many times (2-6 times) eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/stopping/destroying varying quantities of containers for several cycles. Eventually I consistently end up not ever getting ip addresses or being able to even successfully start a container:
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.2.0-40-virtual N/A
linux-backports-modules-3.2.0-40-virtual N/A
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog: |
Using lxc with a btrfs filesystem can lead to system corruption and containers which are unable to boot when many containers are created and started simulatenously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu
`lsb_release -rd`:
ubuntu@ip-10-91-30-131:~$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg05472.html
And here is a key excerpt regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is actually doing. It slowly launches 10 containers and then uses "&" to rapidly fork an additional 10 clone/start operations. I have it doing 2 cycles of this, and running the script many times (2-6 times) eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/stopping/destroying varying quantities of containers for several cycles. Eventually I consistently end up not ever getting ip addresses or being able to even successfully start a container:
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.2.0-40-virtual N/A
linux-backports-modules-3.2.0-40-virtual N/A
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog: |
|
| 2013-08-19 18:39:37 |
Jay Taylor |
description |
Using lxc with a btrfs filesystem can lead to system corruption and containers which are unable to boot when many containers are created and started simulatenously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu
`lsb_release -rd`:
ubuntu@ip-10-91-30-131:~$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg05472.html
And here is a key excerpt regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is actually doing. It slowly launches 10 containers and then uses "&" to rapidly fork an additional 10 clone/start operations. I have it doing 2 cycles of this, and running the script many times (2-6 times) eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/stopping/destroying varying quantities of containers for several cycles. Eventually I consistently end up not ever getting ip addresses or being able to even successfully start a container:
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.2.0-40-virtual N/A
linux-backports-modules-3.2.0-40-virtual N/A
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog: |
Using lxc with a btrfs filesystem can lead to system corruption and containers which are unable to boot when many containers are created and started simultaneously.
Packages involved:
- btrfs-tools
- ubuntu-lxc/daily from http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu
`lsb_release -rd`:
ubuntu@ip-10-91-30-131:~$ lsb_release -rd
Description: Ubuntu 12.04.2 LTS
Release: 12.04
Here is the relevant discussion thread: http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg05472.html
And here is a key excerpt regarding reproducing the problem:
The general system state is something like:
N containers already running happily
Launch N+ more containers in rapid succession (in parallell, not serially).
Here is a test script which closely reflects what my application is actually doing. It slowly launches 10 containers and then uses "&" to rapidly fork an additional 10 clone/start operations. I have it doing 2 cycles of this, and running the script many times (2-6 times) eventually triggers the problem.
test.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
path=/mnt
sudo lxc-destroy -n c1 2>/dev/null
sudo lxc-create -t ubuntu -B btrfs -n c1
for i in `seq 1 10`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 11 20`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
sleep 10
# Create even more.
for i in `seq 21 30`; do
sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i
sudo lxc-start -d -n $prefix$i
done
for i in `seq 31 40`; do
echo $(sudo lxc-clone -s -B btrfs -P $path -o c1 -n $prefix$i; sudo lxc-start -d -n $prefix$i) &
done
stop.sh:
#!/usr/bin/env bash
prefix=$1
test -z "${prefix}" && echo 'error: missing required parameter: prefix' 1>&2 && exit 1
sudo lxc-destroy -n c1;
for i in `seq 1 40`; do
echo $(sudo lxc-stop -k -n $prefix$i; sudo lxc-destroy -n $prefix$i) &
done
bash ./test.sh x
bash ./test.sh y
bash ./test.sh z
If it doesn't manifest at first, try starting/stopping/destroying varying quantities of containers for several cycles. Eventually I consistently end up not ever getting ip addresses or being able to even successfully start a container:
x1 RUNNING - - NO
x10 RUNNING - - NO
x11 RUNNING - - NO
x12 RUNNING - - NO
x13 RUNNING - - NO
x14 RUNNING - - NO
x15 RUNNING - - NO
x16 RUNNING - - NO
x17 RUNNING - - NO
x18 RUNNING - - NO
x19 RUNNING - - NO
x2 RUNNING - - NO
x20 RUNNING - - NO
x21 RUNNING - - NO
x22 RUNNING - - NO
x23 RUNNING - - NO
x24 RUNNING - - NO
x25 RUNNING - - NO
x26 RUNNING - - NO
x27 RUNNING - - NO
x28 RUNNING - - NO
x29 RUNNING - - NO
x3 RUNNING - - NO
x30 RUNNING - - NO
x31 RUNNING - - NO
x32 RUNNING - - NO
x33 RUNNING - - NO
x34 RUNNING - - NO
x35 RUNNING - - NO
x36 RUNNING - - NO
x37 RUNNING - - NO
x38 RUNNING - - NO
x39 RUNNING - - NO
x4 RUNNING - - NO
x40 RUNNING - - NO
x5 RUNNING - - NO
x6 RUNNING - - NO
x7 RUNNING - - NO
x8 RUNNING - - NO
x9 RUNNING - - NO
ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-40-virtual 3.2.0-40.64
ProcVersionSignature: Ubuntu 3.2.0-40.64-virtual 3.2.40
Uname: Linux 3.2.0-40-virtual x86_64
AcpiTables:
AlsaDevices:
total 0
crw-rw---T 1 root audio 116, 1 Aug 14 20:40 seq
crw-rw---T 1 root audio 116, 33 Aug 14 20:40 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: command ['iw', 'reg', 'get'] failed with exit code 1: nl80211 not found.
Date: Mon Aug 19 18:20:19 2013
Ec2AMI: ami-d0f89fb9
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: us-east-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-88aa75e1
Ec2Ramdisk: unavailable
Lspci:
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MarkForUpload: True
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: root=LABEL=cloudimg-rootfs ro console=hvc0
RelatedPackageVersions:
linux-restricted-modules-3.2.0-40-virtual N/A
linux-backports-modules-3.2.0-40-virtual N/A
linux-firmware 1.79.4
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
WifiSyslog: |
|
| 2013-08-19 18:45:42 |
Jay Taylor |
bug |
|
|
added subscriber Serge Hallyn |
| 2013-08-19 19:00:22 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2013-08-19 19:14:37 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
High |
|
| 2013-08-19 19:19:05 |
Joseph Salisbury |
tags |
amd64 apport-bug btrfs ec2-images kernel kernel-bug linux lxc precise |
amd64 apport-bug btrfs ec2-images kernel kernel-bug kernel-da-key linux lxc precise |
|
| 2013-08-19 21:11:46 |
Jay Taylor |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2013-08-24 04:14:41 |
fedorowp |
bug |
|
|
added subscriber fedorowp |
| 2013-09-14 04:36:04 |
sanjayayogi |
bug |
|
|
added subscriber sanjayayogi |
