"Hammering" via ssh crashes Ubuntu Server

[Aero9000]

Binary package hint: ssh

Hi Folks,

I have set up a Ubuntu Server 7.10 and as part of the install I installed the OpenSSH server.

When the server is booting, and I type "ssh user@my-server" on a remote workstation, and keep repeating that while the server is booting, the server drops to an initramfs shell and has to be restarted from the server console. My "workaround", if you will, is to wait 3 minutes after I issued the "reboot" command to make sure the server is ready to accept an incoming ssh connection. But obviously, if for some reason the shutdown procedure takes too long I'm still screwed.

The Server:
Ubuntu 7.10 Server edition
9 GB RAM (yes, nine :-) )
2x 137 GB SCSI disks, coupled via RAID 1
2x dual-core Intel Xeon @ 3 GHz

After I had found this bug I installed Debian 4.0. Debian does NOT have this bug. Right now I'm back on Ubuntu again.

Thanks and best.

[Jamie Strandboge (jdstrand)]

Marked as security vulnerability as it could cause DoS.

[Colin Watson (cjwatson)]

openssh isn't running during the initramfs. I'm not sure what this is, but I have trouble seeing how it could be an openssh bug since none of its code is there.

[Aero9000 (mbg)]

It would *appear* that this bug was addressed by a later update of Ubuntu server. Running at the latest updates via apt-get, I am no longer able to reproduce this bug. So I guess it's no longer high priority...

[Jamie Strandboge (jdstrand)]

I am assuming this is some sort of networking bug in the kernel, and am moving it there. I cannot reproduce this. I tried connecting to a gutsy server and desktop with hundreds of ssh connections with no issues.

Can you reliably reproduce this, and if so how?

[Jamie Strandboge (jdstrand)]

I posted this before reloading the page and didn't see your comment until after.

Since the user reported an up to date system is not affected, I am going to close this bug. Please re-open the bug if it becomes reproducible.

[Aero9000 (mbg)]

Today I had the same error again. This time I was hammering the server with SSH and SSHFS at the same time. Here's what I saw on the console:

Starting up ...
Please wait...
mount: Mounting /dev/disk/by-uuid/<long string> on /root failed: Device or Resource busy
mount: Mounting /root/dev/ on /dev/.static/dev failed: No such file or directory
mount: Mounting /sys on /root/sys failed: No such file or directory
mount: Mounting /proc on /root/proc failed: No such file or directory
Target file system doesn't have /sbin/init

Busybox v1.1.3 (Debian 1:1.1.3-5ubuntu7) Built-in shell (ash)
Enter 'help' for list of built-in commands.

(initramfs)

Best.

[Aero9000 (mbg)]

Please see latest post

[Kees Cook (kees)]

This is a pretty strange failure. Thanks for the screencap. To help narrow this down, I have a bunch of questions:

What filesystem type is your root files?
What are the contents of /proc/mdstat and /proc/partitions on a successful boot?
Does this only happen when you're trying to connect to the server via SSH?
Does it happen on its own if you reboot several times in a row?

Thanks,

[Aero9000 (mbg)]

Hi Kees,

Here are the answers to your questions:
The root file system is ext3.

These are the contents of /proc/mdstat:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sda5[0] sdb5[1]
      135556352 blocks [2/2] [UU]

unused devices: <none>

And /proc/partitions:
major minor #blocks name

   8 0 143374744 sda
   8 1 7815591 sda1
   8 2 1 sda2
   8 5 135556438 sda5
   8 16 143374744 sdb
   8 17 7815591 sdb1
   8 18 1 sdb2
   8 21 135556438 sdb5
   8 32 71833096 sdc
   8 33 71826583 sdc1
   9 0 135556352 md0

Rebooting the server multiple times (I did 5 or 6 reboots) does not lead to the error.

But I do have some other observations...

I suspect this problem is related to running a software raid. When I first started out with this server, I had root in a RAID1 configuration as well. In that configuration, if I hammered the server via SSH, it would always crash. Here, by the way, are also the contents of /proc/mounts:

rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec 0 0
none /proc proc rw,nosuid,nodev,noexec 0 0
udev /dev tmpfs rw 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
/dev/disk/by-uuid/4e63733e-df6c-42e4-8061-8f2597d20e81 / ext3 rw,data=ordered 0 0
/dev/disk/by-uuid/4e63733e-df6c-42e4-8061-8f2597d20e81 /dev/.static/dev ext3 rw,data=ordered 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /dev/shm tmpfs rw 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,noexec 0 0
tmpfs /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
/dev/sdc1 /media/sdc1 ext3 rw,data=ordered 0 0
/dev/md0 /opt ext3 rw,data=ordered 0 0
securityfs /sys/kernel/security securityfs rw 0 0

And the contents of /proc/swaps:
Filename Type Size Used Priority
/dev/sdb1 partition 7815580 0 -1

I hope this helps.

Thanks and best regards,
Maurice.

[Aero9000 (mbg)]

Hi there,

Here's is another (and sure) way to crash the server. During the boot process hammer the server with the keyboard, i.e. repeatedly press a key during the start of Ubuntu (I used Caps Lock). Seems to crash it every time. Just to be sure it wasn't a bad keyboard I had attached, I removed the keyboard, powered off and powered on the server and hammered it via SSH ---> dropped to initramfs. :-(

[Daniel T Chen (crimsun)]

Can you reproduce this bug with a vanilla 2.6.2[4567] kernel from kernel.org?

[Aero9000 (mbg)]

I built a new kernel (2.6.27.8) and on at least one occasion the server dropped into initramfs. This occurred after the server had remained switched off for quite some time (about 4 weeks). It had remained powered, though. At that particular occurrence I was not been hammering the server. In fact, I have not been able to have the server drop in initramfs by hammering it via SSH.

To rule out any hardware related issues, I will re-set up the server with Ubuntu 8.10. Since I am in no real rush to do that, would you have any preference for me to upgrade the existing install or do a fresh install?

Thanks.

[Sergio Zanchetta (primes2h)]

The 18 month support period for Gutsy Gibbon 7.10 has reached its end of life -
http://www.ubuntu.com/news/ubuntu-7.10-eol . As a result, we are closing the
linux-source-2.6.22 kernel task. It would be helpful if you could test the
new Jaunty Jackalope 9.04 release and confirm if this issue remains -
http://www.ubuntu.com/getubuntu/releasenotes/904overview. If the issue still exists with the Jaunty
release, please update this report by changing the Status of the "linux (Ubuntu)"
task from "Incomplete" to "New". Also please be sure to run the command below
which will automatically gather and attach updated debug information to this
report. Thanks in advance.

apport-collect -p linux-image-2.6.28-11-generic 201736

[Jeremy Foshee (jeremyfoshee)]

This bug report was marked as Incomplete and has not had any updated comments for quite some time. As a result this bug is being closed. Please reopen if this is still an issue in the current Ubuntu release http://www.ubuntu.com/getubuntu/download . Also, please be sure to provide any requested information that may have been missing. To reopen the bug, click on the current status under the Status column and change the status back to "New". Thanks.

[This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]