sshd server crashes on startup

[Robert John Bowles]

Please refer to the Ubuntu Forums thread where I sought help on this issue before filing this bug report: http://ubuntuforums.org/showthread.php?t=2015385

This affects two laptops in my possession, both operating under 64-bit Ubuntu 12.04 LTS:
Dell Inspiron M4040 (AMD processor)
Dell Studio XPS 1364 (Intel i7 processor)
A third laptop running 32-bit Ubuntu 12.04 is unaffected.

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: openssh-server 1:5.9p1-5ubuntu1
ProcVersionSignature: Ubuntu 3.2.0-26.41-generic 3.2.19
Uname: Linux 3.2.0-26-generic x86_64
NonfreeKernelModules: fglrx
ApportVersion: 2.0.1-0ubuntu8
Architecture: amd64
Date: Mon Jul 9 12:02:44 2012
InstallationMedia:

SourcePackage: openssh
UpgradeStatus: Upgraded to precise on 2012-05-05 (64 days ago)

[Robert John Bowles (bobjohnbowles)]

This problem started approximately 2 weeks ago. Before then, both laptops were using sshd with no problems. I cannot be exact about the date this started, but this leads me to suspect that a recently-changed 64-bit library package may be at the root.

Also, I have tried using the dropbear package for ssh server, and this appears to fail in a very similar way.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, your crash report is either missing or challenging to deal with as a ".crash" file. Please follow these instructions to have apport report a new bug about your crash that can be dealt with by the automatic retracer.

If you are running the Ubuntu Stable Release you might need to enable apport in /etc/default/apport and restart.

Now open your file manager, navigate to your /var/crash directory and open the crash report you wish to submit.
If this fails you will have to open a terminal and file your report with 'ubuntu-bug /var/crash/_my_crash_report.crash' where _my_crash_report.crash is the crash you would like to report. If you get an error that you aren't allowed to access this report you will have to file it with 'sudo ubuntu-bug /var/crash/_my_crash_report.crash'.

I'm closing this bug report since the process outlined above will automatically open a new bug report which can then dealt with more efficiently. Thanks in advance for your cooperation and understanding.

[Robert John Bowles (bobjohnbowles)]

Hi Robie

I have tried doing as you suggest, but the automatic upload of the
information appears not to be working. Possibly this is another symptom
of the same problem. I also tried implementing the upload from the
command line, but the upload failed there, too.

I am attaching TWO crash reports to this email, one from sshd, and one
from dropbear, both generated in the last few minutes.

I have crash reports on my other computer for dropbear and sshd for last
Friday, but the automatic process failed there because I had removed the
packages. Ironically, when I re-installed openssh-server on that machine
to enable reporting of the crashes, it obligingly worked right away.
Whether this is some kind of fluke or what, I cannot say. I went round
the loop purging and re-installing sshd so many times on Friday, all to
no avail.

So, sshd has started working on one of the computers as mysteriously as
it stopped. On the other computer, sshd, dropbear _and_ ubuntu-bug all
seem to have problems. Crash reports are attached, as they failed to
automatically upload.

Best Wishes Bob Bowles

On 09/07/12 19:21, Robie Basak wrote:
> Thank you for taking the time to report this bug and helping to make
> Ubuntu better. However, your crash report is either missing or
> challenging to deal with as a ".crash" file. Please follow these
> instructions to have apport report a new bug about your crash that can
> be dealt with by the automatic retracer.
>
> If you are running the Ubuntu Stable Release you might need to enable
> apport in /etc/default/apport and restart.
>
> Now open your file manager, navigate to your /var/crash directory and open the crash report you wish to submit.
> If this fails you will have to open a terminal and file your report with 'ubuntu-bug /var/crash/_my_crash_report.crash' where _my_crash_report.crash is the crash you would like to report. If you get an error that you aren't allowed to access this report you will have to file it with 'sudo ubuntu-bug /var/crash/_my_crash_report.crash'.
>
> I'm closing this bug report since the process outlined above will
> automatically open a new bug report which can then dealt with more
> efficiently. Thanks in advance for your cooperation and understanding.
>
> ** Changed in: openssh (Ubuntu)
> Status: New => Incomplete
>

--
Bob Bowles

UK 英国
23 High Street Marsham Aylsham Norfolk NR10 5AE
Tel: [phone] Mob: [phone]

China 中国
China Liaoning Sheng Tieling Shi Yinzhou Qu Shifu Lu Zhong Xing Xiao Qu 9 Hao Lou 2 Dan Yuan 101 Shi 112000
中国辽宁省铁岭市银州区市府路中兴小区九号楼二单元101室 112000
手提电话：[phone]

Skype: bob.john.bowles
Email: <email address hidden>
<email address hidden>

[Robert John Bowles (bobjohnbowles)]

Hi Robie

Please find attached the crash reports from last Friday, from my other
computer. One is for sshd, the other for dropbear.

Best Wishes Bob Bowles

-------- Original Message --------
Subject: Re: [Bug 1022434] Re: sshd server crashes on startup
Date: Mon, 09 Jul 2012 22:17:45 +0800
From: Bob Bowles <email address hidden>
To: Bug 1022434 <email address hidden>

Hi Robie

I have tried doing as you suggest, but the automatic upload of the
information appears not to be working. Possibly this is another symptom
of the same problem. I also tried implementing the upload from the
command line, but the upload failed there, too.

I am attaching TWO crash reports to this email, one from sshd, and one
from dropbear, both generated in the last few minutes.

I have crash reports on my other computer for dropbear and sshd for last
Friday, but the automatic process failed there because I had removed the
packages. Ironically, when I re-installed openssh-server on that machine
to enable reporting of the crashes, it obligingly worked right away.
Whether this is some kind of fluke or what, I cannot say. I went round
the loop purging and re-installing sshd so many times on Friday, all to
no avail.

So, sshd has started working on one of the computers as mysteriously as
it stopped. On the other computer, sshd, dropbear _and_ ubuntu-bug all
seem to have problems. Crash reports are attached, as they failed to
automatically upload.

Best Wishes Bob Bowles

On 09/07/12 19:21, Robie Basak wrote:
> Thank you for taking the time to report this bug and helping to make
> Ubuntu better. However, your crash report is either missing or
> challenging to deal with as a ".crash" file. Please follow these
> instructions to have apport report a new bug about your crash that can
> be dealt with by the automatic retracer.
>
> If you are running the Ubuntu Stable Release you might need to enable
> apport in /etc/default/apport and restart.
>
> Now open your file manager, navigate to your /var/crash directory and open the crash report you wish to submit.
> If this fails you will have to open a terminal and file your report with 'ubuntu-bug /var/crash/_my_crash_report.crash' where _my_crash_report.crash is the crash you would like to report. If you get an error that you aren't allowed to access this report you will have to file it with 'sudo ubuntu-bug /var/crash/_my_crash_report.crash'.
>
> I'm closing this bug report since the process outlined above will
> automatically open a new bug report which can then dealt with more
> efficiently. Thanks in advance for your cooperation and understanding.
>
> ** Changed in: openssh (Ubuntu)
> Status: New => Incomplete
>

--
Bob Bowles

UK 英国
23 High Street Marsham Aylsham Norfolk NR10 5AE
Tel: [phone] Mob: [phone]

China 中国
China Liaoning Sheng Tieling Shi Yinzhou Qu Shifu Lu Zhong Xing Xiao Qu 9 Hao Lou 2 Dan Yuan 101 Shi 112000
中国辽宁省铁岭市银州区市府路中兴小区九号楼二单元101室 112000
手提电话：[phone]

Skype: bob.john.bowles
Email: <email address hidden>
<email address hidden>

[Michael Speth (conzar)]

Here is an error from starting sshd with the -Ddd flag. This happens on a fresh install of 11.10 and also an update from 11.04 to 11.10
*** glibc detected *** /usr/sbin/sshd: double free or corruption (out): 0x00007f8b84edd840 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7a6e6)[0x7f8b81ca86e6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f8b81cac9cc]
/lib/x86_64-linux-gnu/liblsp.so(freeaddrinfo+0x13a)[0x7f8b83931da2]
/usr/sbin/sshd(main+0x1777)[0x7f8b83d654c7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f8b81c4f30d]
/usr/sbin/sshd(+0xf919)[0x7f8b83d66919]

[Robert John Bowles (bobjohnbowles)]

Yep, looks horribly familiar to me.

[Michael Speth (conzar)]

I ran a crash report (https://help.ubuntu.com/community/ReportingBugs) but I don't see my crash report on this bug. Does it need to be accepted?

[Michael Speth (conzar)]

I compiled openssh on my own and the following versions failed with the same error: 4.5p1, 5.8p2, and 6.0p1.

[Robert John Bowles (bobjohnbowles)]

Hi Michael, I have had problems getting the Ubuntu Apport system to work, it seems to cut out before uploading the files. It is not just this bug, either, I have encountered a number of crashes where I go through the process, and then instead of opening the browser page, nada.

The crash reports I have added here have been manually attached to emails I sent to this bug report in response to Robie.

You can also do it on this web page. On the right is a box titled 'Bug attachments', and the last entry is 'Add attachment'.

[Robie Basak (racb)]

Any idea what /lib/x86_64-linux-gnu/liblsp.so is? This seems to be related to the crash, but I can't figure out where this comes from. Is it part of Ubuntu? My search of all files that Ubuntu provides across all packages is coming up empty. Am I doing something wrong, or is this a third party library?

I need debugging symbols from this library to understand the crash dump, but I can't find where to get them since I can't find a package in Ubuntu that provides it. The only Google hit is http://ubuntuforums.org/showthread.php?t=2013205 which looks like a similar crash and was fixed by reinstalling, which perhaps removed liblsp.so?

[Robie Basak (racb)]

It looks like freeaddrinfo is defined in libc.so.6, but on both of your systems it is being overridden by liblsp.so. Interestingly, this file isn't versioned, which is a bit odd. Is this LD_PRELOAD in use perhaps?

Is this override intentional, or is this some kind of malware? Or am I completely missing some other completely innocent reason for this?

[Robert John Bowles (bobjohnbowles)]

Hi Robie

I think you may be onto something, but I don't have the knowledge to offer any concrete way forward. I have seen liblsp.so and libc.so.6 repeatedly referred to in the crash reports I am getting, but this could just be because these libraries are so basic to the system functioning. AFAIK these are part of the core c libraries used in the kernel, they are deep in the system.

FYI sshd is now working on one of my computers, but I am equally puzzled as to why it started working again as to why it failed in the first place. I completely rebuilt the / partition, but this failed on its own to get sshd working. However, a few days and many reboots later, suddenly there it was, working. Possibly something else I downloaded in the interim made the difference, but it is a mystery what or why.

I have attached my /var/log/apt/history.log from the machine in question, in case you can make anything of it.

I don't know if this is a related issue, but as referred to in my message to Michael yesterday, apport or ubuntu-bug does not seem to be working properly either. This complicates the process of getting you the data.

[Robert John Bowles (bobjohnbowles)]

This is just a thought, and I have no real evidence, but the problems started on my systems on or around the time I installed Eclipse and Python 3.2 to do some exploratory software work, just learning my way around Python. I also installed Pygame, which had to be compiled and installed manually to upgrade it to 3.2. Could there be a connection here?

Since I rebuilt the machine that now works, it has all that stuff re-installed, so this really is just a speculation.

[Robert John Bowles (bobjohnbowles)]

Some more experiments following a lead about liblsp.so. This was not (as I thought) part of the core system, but it belongs to the astrill package, which provides vpn client software.

It so happens I was given a beta version of the client. On the machine that mysteriously re-started working with sshd I had removed the beta clent and re-installed an older version.

To confirm, I uninstalled the beta from the 'bad' box, and immediately found sshd worked straight away on reboot. I reinstalled the older version of the Astrill client, and sshd continued to work fine.

The problem, therefore, lies in the astrill package (or at least the beta version I was given). It is not part of the core Ubuntu system, so I am marking this bug 'invalid'. I will take up resolution with the software providers.

[Michael Speth (conzar)]

I also have the Astrill beta client installed! I'll uninstall it after work and try it out!

[Robert John Bowles (bobjohnbowles)]

Looks like we found it!

Just to update you, I have notified Astrill of this issue via their helpdesk. I am awaiting a response. This certainly needs fixing in their code before it goes on general release.

[Michael Speth (conzar)]

Yep. I have completely uninstalled Astrill and now sshd and samba is working!

[Astrill (astrill)]

This isse should be fixed in latest Astrill version. If you still experience problems, please open a ticket directly with Astrill.