ssh server extremly unresponsive

Bug #782092 reported by Martin Konôpka
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

I describe the issue in few steps below.

1.) I want to transfer files from machine A (with ubuntu 11.04, but this is not important) to machine B (ubuntu 10.10; this is important to say) using scp.

2.) Machine B is under constant computational load due to a user program (a simulation) such that most of its CPU power is consumed by the program but it uses less then half of total (12 GB) memory and there is not any significant usage of disk and no other application is running. (Even no user logged into a GNOME session.)

3.) I issue the command scp on machine A; I immediatelly get prompt to give password. However the transfer does not start immediatelly after that. I must wait for some time, say half a minute or even full minute or more (I have not measured it.) just to see the start of the tranfer.

Few additional remarks:

Until recently machine B had had installed ubuntu 10.04 (Lucid Lynx, 64-bit) and there was no such problem even at higher computational loads. Similarly there was no such problem even in more distant past when the machine was running under Linux Mint Helena (based upon Ubuntu 9.10).

The long delays when doing scp occur if machine B was not exposed to an scp or ssh demand for a longer period of time. (Typically I face the problem in the morning when.) If an scp transfer is finally accomplished then I can start a new one which already works without delay.

Machine B is equipped with an Intel i7 CPU, 12 GB of DDR3 and it is a quite powerful workstation. All cores of CPU are set to run at maximum frequency.

I attach a file with content shown by the program top. (Grabbed in situation when a user is logged into GNOME session.)

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: openssh-server 1:5.5p1-4ubuntu5
ProcVersionSignature: Ubuntu 2.6.35-28.50-generic 2.6.35.11
Uname: Linux 2.6.35-28-generic x86_64
Architecture: amd64
Date: Fri May 13 10:39:53 2011
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
SourcePackage: openssh
---
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.23.
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: konopka 12733 F.... pulseaudio
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'Intel'/'HDA Intel at 0xd7ff8000 irq 73'
   Mixer name : 'Analog Devices AD1989B'
   Components : 'HDA:11d4989b,10438372,00100300'
   Controls : 48
   Simple ctrls : 27
DistroRelease: Ubuntu 10.10
HibernationDevice: RESUME=UUID=c49e29ce-51d0-4841-ba53-2277847b7096
InstallationMedia: Ubuntu 10.10 "Maverick Meerkat" - Release amd64 (20101007)
IwConfig:
 lo no wireless extensions.

 eth0 no wireless extensions.

 eth1 no wireless extensions.
MachineType: System manufacturer System Product Name
Package: openssh
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.35-30-generic root=UUID=ed65d6b9-ca7c-4627-808f-b9f882327a13 ro quiet splash
ProcEnviron:
 LANGUAGE=en_GB:en
 PATH=(custom, user)
 LANG=en_GB.utf8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.35-30.54-generic 2.6.35.13
Regression: Yes
RelatedPackageVersions: linux-firmware 1.38.6
Reproducible: Yes
RfKill:

Tags: maverick kernel-config regression-release needs-upstream-testing
Uname: Linux 2.6.35-30-generic x86_64
UserGroups: adm admin cdrom dialout lpadmin plugdev sambashare
WifiSyslog:

dmi.bios.date: 11/12/2009
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 0704
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: P6T DELUXE V2
dmi.board.vendor: ASUSTeK Computer INC.
dmi.board.version: Rev 1.xx
dmi.chassis.asset.tag: Asset-1234567890
dmi.chassis.type: 3
dmi.chassis.vendor: Chassis Manufacture
dmi.chassis.version: Chassis Version
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr0704:bd11/12/2009:svnSystemmanufacturer:pnSystemProductName:pvrSystemVersion:rvnASUSTeKComputerINC.:rnP6TDELUXEV2:rvrRev1.xx:cvnChassisManufacture:ct3:cvrChassisVersion:
dmi.product.name: System Product Name
dmi.product.version: System Version
dmi.sys.vendor: System manufacturer

Revision history for this message
Martin Konôpka (martin.konopka) wrote :
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Martin, first, thanks for taking the time to file this bug report and help us make Ubuntu better.

This sounds more like a bug in the kernel's scheduling than openssh itself, I suspect anything trying to get
scheduled would have problems competing. Opening task against the kernel as well.

If you renice "prog.X" with

sudo renice `pidof prog.X` 5

Does the effect go away, or at least reduce significantly?

Also once you've ssh'd to the machine, do other programs run sluggishly?

Marking openssh task incomplete pending response (If it does go away with renice, and/or all programs that need CPU are sluggish, then this is not an openssh problem IMO).

Changed in openssh (Ubuntu):
status: New → Incomplete
Revision history for this message
Martin Konôpka (martin.konopka) wrote :

Clint, I can now answer your questions, at least partially.

* If I renice the program, the effect is likely gone. However, I will be sure only after a longer period of testing.

* Other programs run swiftly (also without renice-ing prog.x).

I will comment on this issue sometime next week or so, as soon as I acquire more evidence. It may be useful to remind the following info from my orginal report:

The long delays in scp procedure occur if machine B was NOT exposed to an scp or ssh demand for a longer period of time. (Typically I face the problem in the morning.) If an scp transfer is finally accomplished then I can start a new one which already works without a delay.

Revision history for this message
Martin Konôpka (martin.konopka) wrote :

I now have acquired enough evidence and confirm all my writings above.
It implies that there may be a scheduling issue in the linux kernel of ubuntu 10.10.
I will try to check if there is similar issue also with Natty.

(I have 15 years of experience in performing CPU intensive tasks on UNIX/Linux workstations
and servers and this is most likely for the first time when I experience the reported kind
of the odd behaviour.)

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in dianosing the problem. From a terminal window please run:

apport-collect 782092

and then change the status of the bug back to 'New'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Martin Konôpka (martin.konopka) wrote : AcpiTables.txt

apport information

tags: added: apport-collected
description: updated
Revision history for this message
Martin Konôpka (martin.konopka) wrote : AlsaDevices.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : AplayDevices.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : ArecordDevices.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : BootDmesg.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : Card0.Amixer.values.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : Card0.Codecs.codec.0.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : Lspci.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : Lsusb.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : PciMultimedia.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : ProcModules.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : UdevDb.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote : UdevLog.txt

apport information

Revision history for this message
Martin Konôpka (martin.konopka) wrote :

Next weeks I hopefully will acquire more information about the issue, likely also by testing newest kernel. However, due to the nature of the bug, it is possible to test it typically 1 time per day.

Changed in linux (Ubuntu):
status: Incomplete → New
Changed in openssh (Ubuntu):
status: Incomplete → New
Brad Figg (brad-figg)
Changed in linux (Ubuntu):
status: New → Confirmed
Dave Walker (davewalker)
Changed in openssh (Ubuntu):
importance: Undecided → Medium
Revision history for this message
penalvch (penalvch) wrote :

Martin Konopka, thank you for reporting this and helping make Ubuntu better. Maverick reached EOL on April 10, 2012.
Please see this document for currently supported Ubuntu releases:
https://wiki.ubuntu.com/Releases

We were wondering if this is still an issue in a supported release? If so, can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command in a supported release from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux <replace-with-bug-number>

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.

Please let us know your results. Thanks in advance.

tags: added: needs-upstream-testing
Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Martin Konôpka (martin.konopka) wrote :

Greetings,
for now I can at least say that this problem does not occur in ubuntu 11.04 (Natty) running "official" kernels.
I think I have now enough evidence about it both for server and deskop 64-bit kernels in Natty.
My guess is that this would not be an issue also in latest upstream kernel.
Anyway If I find the problem in 12.04 LTS, I will report it and make the tests with the kernels.

Revision history for this message
penalvch (penalvch) wrote :

Martin Konopka, this bug report is being closed due to your last comment regarding this being fixed with an update. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. Thank you again for taking the time to report this bug and helping to make Ubuntu better. Please submit any future bugs you may find.

no longer affects: openssh (Ubuntu)
Changed in linux (Ubuntu):
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.